This is important

My 2025 in music: A review

Tue, 20 Jan 2026 21:55:48 +0000

I’ve written this post for years — comparing Spotify Wrapped with my Last.fm data, exploring how Spotify put together their music data categories and how the results of Spotify Wrapped doesn’t quite capture my music taste, pointing to my Last.fm data instead (I’m especially proud of last year’s deep dive).

It’s still relevant to compare these different patterns, but in reviewing my 2025 in music, the reading and research I’ve been doing for years culminated in some continued realizations.

This past year, I read Liz Pelly’s book Mood Machine and related coverage and commentary, researched in more depth the music metadata that companies like Spotify derive, compute, and compile alongside user behavior data, and followed Hanif Abdurraqib on Instagram (and thus got exposed to his thoughtful cultural criticism on a more frequent basis than his books I dream of my favorite thinkers getting their own website or blog, and then I consider that the reason social media and even Substack are so popular is that the platforms make it so low-friction to share your thoughts, no matter how well (or ill) considered they might be. ).

As part of that reading, I’ve internalized that while frequency of listens is indicative of something, it isn’t necessarily “favorite” or “best”. Repeat listens can be a proxy for “comforting” or “dissociative” or “nonconfrontational” as often as it might be a proxy for “favorite”.

I’ve spent years evaluating my quantitative music listening habits, but it wasn’t until a few years ago that I started actually considering what my favorites of the year were Prompted by a friend who sends out his own favorites every year to a huge email thread full of reply-all sharing. .

With all the work I’ve done in data analysis, I’m even more certain that a quantitative approach isn’t the right one to evaluate what music means to me as a listener. I can devise proxy metrics for “best” — “most frequently played”, “fewest skips”, “most playlisted”, or “most purchased” — but those metrics track activities. Quantitative metrics can’t track intent, and certainly can’t track emotion.

What makes music meaningful to me is crafting memories—having lyrics resonate with your lived experience, feeling the intensity of the dynamics in your soul, having a moment where you sit up and go “wow”. And quantitative data can’t capture those experiences.

So in this annual reflection on my music listening habits, let me color in the quantitative data lines a bit more than usual. Let’s review…

My year in music

I listened to an estimated 33,000 minutes¹ of music in 2025. I calculate this estimate by using either the actual track duration of a song (if I own it) or a constant that is based on the average song length in my library (about 4 minutes).

4,449 of those minutes were on Spotify, 13,715 of them on SoundCloud.

What helps that difference along is that I purchased 1,091 songs in 2025 on iTunes and Bandcamp. A few more might have snuck in through Beatport as well, but the metadata traces don’t leak as reliably for those tracks Bandcamp tracks auto-add a comment with a URL to the bandcamp page where the track was purchased, and tracks purchased from the iTunes store have explicit isPurchased metadata. As far as I can tell, no signifier is present in the metadata for tracks purchased from Beatport. , so I can’t count them.

I also listened to some music live, but given the year I had, it’s perhaps not surprising that I only went to 5 concerts and DJ sets in 2025, and only went for the opener once. Recovering from major surgery is no joke (even though it was planned!), and I’m glad to be back on my feet and going to shows.

I listened to 7,968 songs (4,299 unique tracks) and 2,057 total artists, of which 700 were new to me.

Of course, because I don’t run a pristinely clean music metadata pipeline, muddled in those 700 “new” artists are a lot of artists I’ve already discovered. For example, some discovered artists are “Malugi & Wolters” or “BICEP, ELIZA” or “Diffrent | Boiler Room”. If my metadata were cleaner (more accurate), those artists wouldn’t be listed as new artists, because I discovered Malugi in 2024, Wolters in 2024, Bicep in 2017, and Diffrent in 2024.

Among those 700 “new” artists, though, there are some serious highlights that were actually new to me in 2025: Faster Horses, Djo, JADE, GLOCKTA, Joshua Idehen, and IN PARALLEL (it’s trendy to be all caps I guess).

That’s a high-level review of some high-level stats, but let’s get more granular. Who were my top artists of the year?

My “top artists” of 2025

Due to the disparity of listening volume in each place, my top artists look wildly different across the services…

Spotify	SoundCloud	My Last.fm data
HAAi	Beyoncé	Chappell Roan
Bon Iver	Rinse FM	Carly Rae Jepsen
Joe Goddard	Fred again..	Mk.gee
Sabrina Carpenter	Mk.gee	Waxahatchee
Upper90	Tom VR	Tourist

Much like last year, SoundCloud reveals a flaw in its metadata, ranking an online radio service as my second top artist of the year. Shoutout Rinse FM DJ sets!

Comparing Last.fm to Last.fm?

Because I’m writing this post later than usual, I can compare my Last.fm Last.year report with Last.fm stats that I pipe into a Splunk instance. My data pipeline pulling data from the Last.fm API isn’t perfect, so I was anticipating some minimal disparities, but I was very surprised:

Last.fm Last.year	My Last.fm data
The Format	Chappell Roan
Carly Rae Jepsen	Carly Rae Jepsen
Mk.gee	Mk.gee
Chappell Roan	Waxahatchee
Waxahatchee	Tourist

The Format dominated my report and was far-and-away the top artist of my year.

I was perplexed until I noticed this heat map visualization:

Somehow, the data tracking recorded 1,023 music listens (what Last.fm calls scrobbles) for that one day, October 26, 2025. Given that there are 1440 minutes in a day, I definitely slept that day, and the average length of a song in my library is almost 4 minutes, there’s a lot that points to a data error.

I thought maybe the excessive recorded listens was due to a time zone issue caused by some travel, but I didn’t travel on that day. Me being me, I dug deeper.

Visualizing the listening events for the day in question, the issue is obvious. Here’s what a typical listening session On December 8, 2025, I listened to some tracks from Black British Music by Jim Legxacy. looks like as a line chart, plotted by minute:

By contrast, here’s what my listening session looked like on October 26 as recorded in Last.fm:

There’s still 1 track being recorded at a given time, but despite the time bins only being one minute long in the line chart, the lines overlap.

Calculating the differences between events over time reveals that throughout the entire day, 52 events were separated by only 1 second from another event, 45 only 2 seconds, 44 events happened just 3 seconds apart from another event, on down the line:

It’s absurdly inaccurate. As one last example of the data issue, in my iTunes library I have The Format album Dog Problems, The Format album Interventions & Lullabies, and the Snails EP.

I painstakingly reviewed 971 rows of data to attempt to reconstruct a legitimate listening session using timestamps for each recorded listen and track duration. In this reconstruction, I assume that I listened to the tracks of their discography in my iTunes library once consecutively.

Here’s how that listening session looks in the data, where each row number corresponds to the event number out of that 971 event period that occurred over a few hours:

Row number	Time	Track name	Album	Track number
1	18:49:24	Matches	Dog Problems	1
69	18:51:46	I’m Actual	Dog Problems	2
131	18:53:58	Time Bomb	Dog Problems	3
196	18:57:59	She Doesn’t Get It	Dog Problems	4
247	19:02:11	Pick Me Up	Dog Problems	5
294	19:04:33	Dog Problems	Dog Problems	6
333	19:08:57	Oceans	Dog Problems	7
365	19:14:42	Dead End	Dog Problems	8
389	19:18:27	Snails	Dog Problems	9
430	19:21:28	The Compromise	Dog Problems	10
459	19:34:01	Inches & Failing	Dog Problems	11
489	19:37:33	If Work Permits	Dog Problems	12
517	21:16:47	The First Single (Cause A Scene)	Interventions & Lullabies	1
527	19:46:43	Wait, Wait, Wait	Interventions & Lullabies	2
569	19:51:05	Give It Up	Interventions & Lullabies	3
594	20:00:58	Tie the Rope	Interventions & Lullabies	4
648	20:02:49	Tune Out	Interventions & Lullabies	5
671	20:09:29	I’m Ready, I Am	Interventions & Lullabies	6
700	20:11:30	Sore Thumb	Interventions & Lullabies	8
728	20:28:18	A Mess to Be Made	Interventions & Lullabies	9
756	20:31:28	Let’s Make This Moment a Crime	Interventions & Lullabies	10
789	20:35:50	Career Day	Interventions & Lullabies	11
803	20:43:12	A Save Situation	Interventions & Lullabies	12
880	20:44:59	Janet	Snails EP	1
907	21:30:17	Wait Wait Wait (Acoustic)	Snails EP	2
925	21:45:18	Tune Out (Acoustic)	Snails EP	3
949	21:50:27	On Your Porch (Acoustic)	Snails EP	4

I italicized The First Single (Cause A Scene) because it is the only instance of that track in the listening history, but it didn’t match up with the listening session I reconstructed.

Buried in this dataset is 68 total recorded listens of the song Janet, and 69 of the track She Doesn’t Get It, ostensibly all in the same 6-hour time span. They’re both excellent tracks, and I’m a big fan of the Format, but their music isn’t good enough to transcend space and time.

They did announce a new album in 2025, and a tour that (finally) replaces their tour originally scheduled for 2020, which is why I was listening to them.

As for the cause of the data anomaly? My only and best guess is that I might have been trying to listen to the tracks on two of my Sonos speakers, and either the app or something about the Airplay connection tripped up the Scrobbles for Last.fm app.

Flukes with data collection can lead to frustrating outliers that can skew an annual analysis of quantitative music listening data, but other flaws are inherent in the form itself.

Flaws of an annual quantitative data review

Quantitative listening data prioritizes volume of listens over time, and a year is a somewhat arbitrary boundary when music discovery happens year round.

For an artist to show up in an end-of-year top 5 ranking based on total listen count, a few things bias an artist toward inclusion:

Early discovery — An artist discovered in January is more likely to have more listens than one discovered in November, making it more likely to be a “top artist” of the year.
Large discography — If an artist has a lot of tracks released, it’s easier to listen to a lot of music by that artist (because there’s a lot to listen to!)
Intent — It’s difficult to listen to an artist a lot by accident, so there usually needs to be some sort of intent, whether it’s to check out the latest album by an artist (and then realize you zoned out, so you put it on again), listen to the back catalog of an artist you just discovered, or put on something familiar while you’re working.

What you spend a lot of time listening to, though, isn’t necessarily what you most enjoyed listening to, or what had an emotional or other impact. It’s easier to listen to things and forget about them and drift into the sameness, and then that artist shows up on your most-listened-to list at the end of the year because you put their discography on while you completed an all-day task.

Often, the songs I most enjoy listening to are the ones where, when they come on, I mentally picture myself at a club dancing to them, or sing along to them in my head, or pull them up so that I remember what the track is so that I can recommend it to a few friends later. They’re the ones that make me sit up and take notice.

That isn’t to say that the artists I listen to in the background aren’t good, or that I don’t enjoy them — I have to, in order to listen to them that much. I need them to be good! But at the level of frequency of listens needed to break into the top 5 list for me, I’m really listening to them a lot.

The point I’m trying to make here is that frequency of listens can be a proxy for most enjoyed, but it is an imperfect proxy metric for an unclearly defined “objective function” of “best”. That’s why companies like Spotify choose “top”, because top is a clear statistical measure — given a metric, list the top 5 entries in the list.

My top albums of the year

I’m not much of an album listener, but I’m learning! As such, here are my top albums of the year according to Spotify and Last.fm (SoundCloud’s “set” based format doesn’t lend itself that well to tracking albums):

Spotify	My Last.fm data
Global Underground #46: ANNA by ANNA	Two Star & The Dream Police by Mk.Gee
Sunshine on Leith by the Proclaimers	Emotion (Deluxe Expanded Edition) by Carly Rae Jepsen
SABLE, fABLE by Bon Iver	Tigers Blood by Waxahatchee
Harmonics by Joe Goddard	The Rise and Fall of a Midwest Princess by Chappell Roan
Demos by orbit	Portrait of a Legend: 1951–1964 by Sam Cooke

The Spotify albums are fascinating to me. Because of my low listening volume on that service in 2025, the list is a very odd cross-section of listening behavior:

I have literally no recollection of listening to the album that Spotify claims was my top album of the year.
The Proclaimers have some bangers, so I probably put that album on at work one day.
SABLE, fABLE was released in 2025, so this was likely a case of album evaluation while trying to decide whether to purchase it.
Harmonics I first discovered on Spotify after a friend of mine shared it as part of her end-of-year playlist for 2024.
Similarly, Demos by orbit was shared with a friend of mine who was recommending it to me (but I wasn’t a fan, sorry!)

Meanwhile, my top albums of the year reflect some serious obsessions…

Mk.gee was an early 2025 discovery, and the album is short and easy to listen to in terms of suitable moods and temperaments, so I put it on a lot in the early months of the year.
It was the 10th anniversary of Carly Rae Jepsen’s EMOTION and it remains a no skips album. I reveled in the anniversary with some serious on-repeat binge listens.
Tigers Blood reminded me that I can be an indie folk girlie when I feel like it. I ended up seeing Waxhatchee live in 2025 which helped me remember this album again later in the year.
Chappell Roan remains a favorite (I cannot stop listening to her Bonnaroo set on YouTube) so while I’m a little surprised I listened enough for The Rise and Fall of a Midwest Princess to rank fourth on this list, I’m not that surprised.
Portraits of a Legend: 1951 – 1964 is the Sam Cooke greatest hits album and I can’t recommend it enough.

Those were my most-listened-to albums of the year, but there’s basically no overlap with my personal favorite albums (and EPs) of 2025…

My favorite albums of the year

The timing of some of these album releases would’ve made it impossible for them to chart as a “top” album of the year, and this is why it’s relevant to consider different personal metrics besides “top”.

Tourist = Music is Invisible

I’m always here for a Tourist album, and when he promised a trance album I was sold. Classic Tourist but more high energy than usual (as one would expect from a trance album). There’s something about the familiar beats and grooves that earmark his tracks that make me happy and entrance me.
HAAi = HUMANiSE

I didn’t realize how badly I needed this album until her single Can’t Stand to Lose came out, and then I was like oh this is going to be a peaceful, contemplative, gift. After Always Ascending came out a few years ago I am delighted that she collaborated with Jon Hopkins on this album.
Bicep = CHROMA 000

Bicep’s CHROMA era is a bit crunchier than in the past, but it hit. 004 ROLA is maybe my favorite track off the album. Listening to a Bicep album, I can’t help but think of their live sets and how all-encompassing and absorbing it is. The energy of this live album evokes their live set, and I’m grateful for it.
SWIM = Dear Friend

Based on how much I listened to his album In Circles, I was excited for this album. It’s a bit softer in energy than In Circles, but still a solid album with a bunch of favorites.
This list was originally 5 albums long, because Daphni = Butterfly was just good enough to make this list, and then I realized it doesn’t actually fully release until February. 2026 favorite albums list, here I come.

A couple honorable mentions are Skin on Skin = Home is True and Marlon Hoffstadt = All Yours.

Two albums I discovered in the last few weeks also deserve honorable mentions, Djo = The Crux and Jim Legxacy = black british music (2025).

Very different albums, but both are great listens straight through. Jim Legxacy had a song recommended on the Said the Gramophone list for 2025, then I listened to the whole album and was sold. It’s firmly a no skips album.

I thought I discovered Djo and his album The Crux through the Said the Gramophone list too, but after some sleuthing I realized that it must have come from listening to my SoundCloud Playback playlist, which had a bunch of Mk.gee tracks on it. The autoplay after that playlist ended included some Djo tracks, which were so good I dug deeper and here we are. (the same or similar autoplay session led me to discover Florence Road’s track Goodnight, which is a banger)

Favorite EPs of the year

EPs sit in a weird spot. Too long to be a proper single, too short to be an album (although a 7 track album also feels too short to be an album!), I still want to recognize the progression toward a form. Here are my favorites of 2025:

Effy = The Syndicate

Effy crushes this EP with 3 straight bangers. One of the best in the game atm.
DJ HEARTSTRING & SWIM = FOREVER

Another collaboration between two favorites with a really solid set of songs. Hard to go wrong, especially when the opening track was one I’d been wanting to pick up ever since I heard it in a DJ set.
Upper90 & Baron von Trax = At Night EP

They themselves call it an EP but it’s only two tracks! But both are bangers, and it’s only fair for Upper90 to make an appearance in a list of my favorites of the year.
Villager = Need Right / Dogma

Another two-track EP (is it a single at that rate? who even knows with digital releases), both tracks hit hard but Need Right is my favorite to listen to. Dogma is maybe spookier.
X CLUB. = Stay With Me

Something about this EP feels almost retro? Like it didn’t come out in 2025? I can’t really explain it, but Desire To Stay is my favorite track here.
Cameo Blush = Grace in Motion

The best track is Here, but Essa is a close second. The driving background sounds, Cameo Blush is another producer with some signature sounds.

I made up for the list of 4 favorite albums with a list of 6 favorite EPs. I think it’s also telling that the list of favorite EPs is primarily the newer producers on the scene, still finding their sound and crafting their own tracks, whereas the electronic artists on the favorite albums list have been around the block a bit.

My top songs of the year

According to each service that I use to listen to music, and about which I also have quantitative listening behavior data, my top 5 songs of 2025 were as follows:

Spotify	SoundCloud	My Last.fm data
Simon & Garfunkel - [Cecilia]²	Upper90 at Intercell Indoor 2024	Paravi - Angry
HAAi - Can’t Stand To Lose	Upper90 at Glitch Berlin	Faster Horses - Wish U Were Mine <3
The Proclaimers - Then I Met You	Jacques Greene 11th January Rinse FM	KH - HandsToMyself
Joe Goddard - New World (Flow)	LFE-KLUB Mix w/ Upper90	Mk.gee - Alesis
Nina Sky - Move Ya Body	SWU FM: t e s t p r e s s and Baron Von Trax	HAAi - Can’t Stand To Lose

I listened to Cecilia 5 times, which was enough for Spotify to call it my top song of the year.

SoundCloud being SoundCloud, all of my top 5 songs of the year were hour+ long DJ sets, with the Upper90 set from Intercell Indoor 2024 remaining a favorite for focus time at work and a plethora of IDs:

It’s not surprising that HAAi’s track Can’t Stand to Lose made it onto multiple lists. I’d been eagerly awaiting her album and it was the first single released, and therefore had the most opportunity to be listened to repeatedly.

Most of the songs on my Last.fm list either resonated with me at particular moments, or made it onto my quarterly playlists consistently — HandsToMyself was on 3 out of 4 of the quarterly playlists I used as my default listening playlists throughout the year.

The playlist as codifier

What I spend the most time listening to are whichever songs I put on my quarterly playlists. For the most part I’m mindlessly putting on a playlist for my morning commute, so as I buy music it has to make it through the funnel.

There are certainly some artists and albums that I discovered or even purchased music from in 2025 that are great but which I didn’t listen to that much. Looking at my library, Mild Minds’ album GEMINI comes to mind, as does SHERELLE’s WITH A VENGEANCE or BAMBII’s INFINITY CLUB II.

For the artists and albums that didn’t make it to purchase, or didn’t make it to a playlist, the sound might not have found me at the right time or mood, or I was simply too busy to spend time with the music after buying it, and never went back when I did have time.

For some tracks, I liked it enough to buy it but then forgot entirely about it.

Because my rating system requires some input from me — either taking the three-tap step on mobile to rate a track, favorite it, or add it to my quarterly playlist — it can be easy for me to forget that I like a particular artist or track if I don’t take action to record it.

Overall, I listened to 4299 unique songs (and sets) in 2025, purchased 1091 songs, added 309 of those songs to my quarterly playlists, and rated 213 songs.

To visually capture the difference in quantity throughout those different steps, I made this to-scale but pseudo treemap chart by hand:

If a song makes it through to a playlist, odds are that it will stick around and end up on one of these year-end lists!

My top songs of 2025 are all good songs (or sets, as the case may be), but my favorites of the year were much tougher to narrow down.

Favorite tracks of the year

I had 30 favorite tracks of 2025, but I want to talk about 5 that were particularly meaningful, memorable, and enjoyable tracks for me…

Sam Cooke - (What A) Wonderful World

I listen to a podcast called Hit Parade and he had a couple episodes on Sam Cooke, Don’t Know Much About History that I listened to while I was recovering from surgery. I was familiar with his songs You Send Me and Bring it On Home to Me, but hadn’t explored his back catalog. And boy what a history, what a performer, and more. This track is my favorite of the new-to-me tracks, but I recommend the full greatest hits and beyond. I also listened to all of this when I was doing some research into the background of Motown Records, and it was a nice confluence to learn a lot about another soul legend.
Joshua Idehen - It Always Was

From the artist that made “Mum Does The Washing” which featured on 2024’s Said the Gramophone list, this caught my attention immediately with its upbeat and hopeful mantras. The album title also hits: “I know you’re hurting, everyone is hurting, everyone is trying, you have got to try.”
Too $hort - Blow the Whistle (black panda x seanathan remix)

This track is the far and away standout of an 80+ track compilation of bootlegs that has changed the way I hear this song. I can never hear the original again, it’s too slow and ruined for me. A good bootleg/remix is transformational like that.
Glockta - Ricochet

ec2a put out this EP and as a label that has released Soul Mass Transit System, Silva Bumpa, and Club Angel, I try to listen to the releases by artists I don’t recognize when I have the capacity. This track caught my attention immediately and it still has me in its clutches. Some energy from idk, is it drill? Is it something else rap-adjacent? Idk genres anymore but the insistent rhythmic vocals are basically another instrument that keep me moving.

This song has <1000 streams on Spotify, so if you use that service maybe try to drive it up a bit so Glockta gets some royalty cuts? (Or buy it on Bandcamp so they get paid for real).
The Shapeshifters - lolas theme (Luxusvilla edit)

This is the sort of track that I could only discover organically — I first heard this in a trance DJ set (Upper90? Marlon Hoffstadt? It’s lost to me now) but I went down the pathway of digging through comments and trying to find the ID, then trying to find a way to purchase the track. Like most tracks of this length, it takes a bit to get going (probably because it’s meant to be played out, and not at home) but give it a minute (or skip ahead) and it will start to hit.

Even in attempting to rank this list, I couldn’t decide on one definition of “best”. Because I listen to different tracks for different reasons, to fit different purposes, different moods, tasks, and energies, there can be no one best song.

A lot of my favorite tracks of 2025 are high quality tracks from artists that I’ve enjoyed for several years now:

Others were new-to-me artists or tracks that I discovered from DJ sets:

Still more favorites were more organic discoveries from researching upcoming shows happening in the city: Funk Tribu - I Got it For You, or a record label that I follow: Field Motion - Deflate Me.

It’s somewhat chaotic to listen to all of these tracks together (I am primarily still a playlist listener, so I do). A lot can happen in a year and a favorites ranking must also be similarly diverse.

Beyond top artists, top albums, and top songs, Spotify and SoundCloud shared some other information in their 2025 wrap-up posts…

What is Spotify Wrapped for?

In addition to top songs, top artists, and top albums, Spotify usually includes other bits of information that it packages up in an effort to share ~ * insights * ~ about your music listening.

In my opinion, the 2025 Spotify Wrapped really exposed how the company thinks of the users of the Spotify service as consumers of Spotify rather than music.

The excess insights beyond the top songs, artists, albums, and genres focused on your behavior on Spotify used as a proxy for your music listening habits. When you don’t use Spotify much, this perspective falls flat and reveals itself as a transparent skin over the internal advertising-relevant user clusters that Spotify maintains about its users.

For example, let’s dig into the “clubs” and the “listening age” that supplemented the standard list of top artists, songs, and genres in 2025.

Why didn’t the “clubs” land?

In 2025, Spotify Wrapped included some “clubs” that Spotify grouped everyone into. Among the folks I follow online and the online communities I’m part of, I think maybe one person shared their club membership out of dozens that were sharing their Spotify Wrapped stats generally.

So why didn’t the clubs resonate with people? Why did basically no one share their club membership and “role”?

In my opinion, it’s because they were boring and impersonal. There was nothing inspiring or revealing about the club definitions (or your assigned role within a club) — no insights to be gained about yourself. Nothing about the club or membership evoked a sense of personal identity.

I want to explain more about why by digging into the club and role definitions.

According to a Spotify blog post “Join Your Wrapped Club”, published December 7, 2025, the clubs were:

Club	Description
Cloud State Society	Your club finds peace through music. New members are gifted house plants.
Grit Collective	Your club believes in rebellion through music. There is no club rulebook.
Club Serotonin	Your club thrives on good vibes. Members insist on natural lighting.
Full Charge Crew	Your club is committed to the endless party. Club events are high energy, zero chill.
Cosmic Stereo Club	Your club traverses terrestrial barriers with music. Clubhouse seating is entirely bean bags.
Soft Hearts Club	Your club believes that vulnerability is the key to great music. Its doors are always open.

Within each club, Spotify users were also assigned roles that they played within each club:

Role	Description
Leader	Your listening is strongly aligned with club values, making you a perfect role model.
Scout	You listen to the freshest releases, always pushing your club forward.
Archivist	Your listening delves into past eras, ensuring club history never fades.
Curator	You’re a focused playlist creator, combining the best of your club into mixes.
Collector	You often save music to your library, building a large club collection.
Recruiter	You share music far and wide, bringing in frequent new club members.
Loyalist	You rarely skip tracks, confirming your unwavering dedication to the club.
Supporter	Your listening favors one artist, ensuring they’re heard around your club.
Broadcaster	You listen to podcasts more than others, keeping club conversation alive.
Specialist	You explore experimental sounds, refining your club’s sonic boundaries.

What does it all mean?! It’s a lot of word salad without a lot that resonates.

My club was the “Full Charge Crew”, which as described basically says “you listened to a lot of high energy music”. Okay! I did do that. Is the high energy music part of my personal identity? Does it reflect my friend group? (Spotify doesn’t know, becuase it isn’t a social network (although they did recently add messaging back into their app and are probably trying to bring that back I was fairly bullish on Spotify’s collaborative interactions and messaging functionality when I was writing about music discovery patterns back in 2019, but Spotify previously removed messaging back in 2017, and I wrote about that in my music discovery blog post last year. ).)

Instead, my club assignment directly reflects a cluster that Spotify’s machine learning algorithms grouped me into based on my user behavior data collected while I used the service.

As outlined by Spotify in the same blog post:

Your Club captures the emotional qualities of your year in music. That includes the moods, genres, and musical terms that defined what you listened to most.
Each track is tagged with descriptors. Take your favorite breakup anthem, for example. It might be tagged with descriptors such as “heartbreak” and “yearning” because of the titles of the user playlists that it appears on.
These descriptors roll up into six possible Clubs. For each one, we calculate a score based on streams and assign you to the Club with the highest score.
Your role in a Club is determined by your standout on-platform behavior and listening compared to the rest of your Club.

To restate it more technically: Spotify collects cultural metadata about tracks (such as from the titles of playlists that you and others create on the platform). To define the clubs, Spotify grouped the metadata into 6 clusters based on the descriptors. My club (Full Charge Crew), for example, probably had a number of tags like “energy” or “high energy” or “no chill party”.

When each user’s listening patterns were analyzed, the tracks they listened to were scored according to which cluster the tracks belonged to, and the cluster in which you streamed the most songs defined your club. Club is a good name for it, representing both a group of people, and a place you might go to listen to music.

Role assignment, meanwhile, is defined in relation to the club, which means that your role reflects your user behavior on Spotify in relation to the aggregate user behavior patterns of the other users grouped into your club (not your friends, but mysterious strangers that listen like you).

These roles transparently reveal: “This is how you used Spotify in 2025”. The roles and clubs are cold, impersonal, and expose the volume of user data that Spotify collects about its users—primarily to sell ads, but also to fund the development of royalty-free or low-royalty content.

In my opinion, the roles can be basically summarized as:

Role	Honest description
Leader	You listened to music that was mostly the same as other people in your cluster. Good for you! It might be boring, but we’ll call you a leader for doing what everyone else did.
Scout	You listened to new music when it came out. Thanks for that.
Archivist	You listen to music that is older (what the industry calls catalog music, or music released more than 18 months ago). That’s cool.
Curator	You make playlists. Who knows why you make playlists, or what of, or who for, but that’s what we know about you — you make playlists.
Collector	You save music to your library. That’s all we know. You hit that <3 button that we changed into a + button that means different things in different places in the app!
Recruiter	You share music. Click that share button, copy those links. Thanks for sending people to Spotify, you’re recruiting people to our service and driving traffic.
Loyalist	You don’t hit skip very often. This is the most interesting thing that we could identify about your listening behavior. Maybe you listen on a smart speaker, or you work at a gym or a coffee shop that is (grayly) using your Spotify account to play music during the day. Either way, thanks for being Loyal to Spotify.
Supporter	You mostly listen to one artist. How INTERESTING.
Broadcaster	You listen to podcasts more often than other people in the cluster that we arbitrarily grouped you into. This probably means we can sell you more ads, or more specific ads because we have more content available about your interests as a result.
Specialist	You explore “experimental sounds”, which is undefined and seemingly meaningless, but it also makes you a “specialist” but a specialist in… exploration, I guess?

The entire construct focuses on defining you by your behavior inside of Spotify, instead of your engagement with the music itself. Even the roles that are specific to music are defined in contrast to the behavior of other aggregate users in your cluster (sorry, club).

In the past, Spotify has grouped users into clusters as part of Wrapped. Most notably in 2023, identifying streaming habits based on how you use the service, but still framing the content in the context of you — making it more personal that you listen to music on shuffle or full albums or more. The streaming habit is positioned in contrast to other Spotify users.

Meanwhile, the clubs and roles identify other Spotify users and places you with them — assigning you a club and a role that you play in that club that you unwittingly attended by using Spotify.

In 2024, the “music evolution” was fully centered around you and your music listening habits, again drawing aggressively from the cultural metadata about music that Spotify collects, and in 2022 the focus was on a listening personality and an audio day — again, highly personal. Before 2022, Spotify shared more of its music metadata than user behavior data necessarily, focusing on an “audio aura” in 2021.

Instead of revealing anything insightful about how you interact with music, or what you listened to, your club and your role are about how you used Spotify, and what the streaming service is to you. But a club and role assignment wasn’t the only “nonstandard” information Spotify shared—they also assigned users a “listening age”.

What is a listening age, and why did it land with users?

In contrast to clubs, listening age really landed. Most of my friends that shared a Spotify Wrapped post shared their listening age — some people only shared their listening age!

Something about the listening age made it well-suited for the type of virality historically associated with Spotify Wrapped:

It’s somewhat inscrutable. What does it even mean?!
It’s amusing to be told that your age is different than it is (sometimes wildly)

If there’s anything social media has taught us, it’s that dissonance is viral. Agreement doesn’t trend, sameness isn’t attention-grabbing. A listening age that is distant from the actual age of the person consuming the music is perfect for the algorithmically defined internet — the statistic is exceptionally shareable.

Some key examples that came across my feeds:

A friend in her 50s and another friend in her late 40s — both with teenage listening ages.
Another friend in her late 40s and another in her mid 30s — both early 20s listening ages.
One friend in her mid-30s? A listening age in the mid-50s.
More friends in their mid-30s had listening ages in the mid-70s and later, up to age 84.

So how was it calculated? According to Spotify in “2025 Wrapped: The Simple Truth About How Your Wrapped Comes to Life”

Your Listening Age is based on the idea of a “reminiscence bump,” which is the tendency to feel most connected to the music from your younger years.

First, we look at the release dates of all of the songs you played this year.

Next, we identify the five-year span of music that you engaged with more than other listeners your age.

We’re hypothesizing that this five-year span matches your “reminiscence bump,” assuming you were between 16 and 21 years old when those tracks were released.

For example, if you listen to way more music from the late 1970s than others your age, we playfully hypothesize that your “listening” age is 63 today, the age of someone who would have been in their formative years in the late 1970s.

To receive listening age, we needed a reported date of birth in your account of 1925 or more recent, plus 5 streams within a 5-year release date band.

Similar to the clubs, Spotify is again exposing the clusters that they use to group users — in this case, clusters by age, and comparing your behavior against that of others your age.

Spotify knows your real age, so it is indeed a “playful hypothesis” that the “five-year span of music that you engaged with more than other listeners your age” (emphasis added) is a time period that you’re reminiscing about, instead of one that you dove deep into one afternoon (or for several afternoons).

In my case, I listened to a good amount of Sam Cooke on Spotify before purchasing his greatest hits album. Sam Cooke’s song (What a) Wonderful World was released in 1960, my listening age was 78 — because if I was 17 in 1960 (and thus in peak “reminiscence age”), I would be 78 in 2025.

This graphic from Instagram account @_sportsball helps capture the math in chart form, comparing the number of songs that you listened to for a specific year in the yellow bars and the average number of songs from that year listened to by others in your age cluster as the green dots.

The computation of listening age might be playful, but it reinforces the notion that your music taste is set in your teenage years, and that you spend the rest of your life listening to the same music that you did as a teenager, reminiscing about those “brain-shifting listening experiences” that you no longer have.

And Spotify can profit off of this notion. If you are a passive, lean-back music listener, accepting the content that Spotify dishes up to you in the form of algorithmically curated playlists, Spotify can identify what music was popular during your “reminiscence age”, collect & contract a large amount of “perfect fit” content that requires paying low or no royalties, and direct your attention toward that. There is more profit in providing you more of what you might reminisce about than there is in driving true discovery.

What about SoundCloud Playback?

Spotify isn’t the only music service that I use, and SoundCloud also provides a list of top 5 metadata moments of the year. In addition to top artists, songs, and genres, in 2025, SoundCloud shared “top moods”.

For the first time, SoundCloud is exposing more of its slippery cultural metadata. SoundCloud has different opportunities to collect metadata than Spotify does — you can comment on individual tracks, even at specific timestamps—in addition to naming sets (playlists), which is the primary source of on-service cultural metadata at Spotify.

What amused me about my top moods is that in an effort to list 5 for consistency, it listed values that were wildly off scale from each other:

Or to graph those listen volumes (presumably that’s what those numbers are):

At that point, just tell me I had a high energy year in music and call it a day.

I am grateful for this peek behind the metadata curtain, however. SoundCloud doesn’t have nearly as many patents surrounding their music metadata collection, management, and usage as Spotify does (to my chagrin), which is one of the reasons that most of my metadata deep dives have focused on Spotify’s creation and management of it (I also focus on Spotify because they have so much of it).

In addition to far fewer patents, SoundCloud also doesn’t have a truly public API that could reveal some aspects of how they think about their product and what they provide I have a blog post draft I haven’t touched in almost 6 years that focuses on what the API endpoints that a company makes available says about the company and how they think about their product on the market—which customers are they targeting? What does the company expect the consumers of their APIs to do with the data they provide? It’s an insightful exercise to consider Spotify and SoundCloud at different poles of this market. .

While I haven’t dug into SoundCloud’s metadata culture as extensively as that of Spotify’s, I did listen to a HonestQuo Ventures podcast interview with Hazel Savage, their former VP of Music Intelligence and cofounder of (acquired) startup Musiio, which provides automated music metadata tagging.

Given all the investment and clear instances of cultural metadata present on their platform, it’s somewhat sad that we only see “moods” and “genres” revealed in the SoundCloud Playback. Perhaps there will be more peeks behind the curtain in the future—or I’ll just have to resign myself to the fact that SoundCloud is focusing more efforts on using the metadata to inform a superior recommendation algorithm rather than a copycat marketing stunt.

What exists beyond the streaming picture?

If I consider the picture that Spotify Wrapped paints about my year in music, it looks something like this:

It’s kind of cramped, not very aesthetic, and is unbalanced.

If I add my SoundCloud playback to the equation, it might look more like this…

The SoundCloud picture on its own is more pleasing to experience, but is cramped and lacks a point of focus and visual interest. It’s a rather incomplete picture, but it has more of a lens into the full picture.

The size of these boxes is non-representative, but rather metaphorical. And what isn’t captured in this figure is how it felt to be sitting at that alpine lake, relaxing on a rock with my partner and enjoying the fact that I had managed to do the hike at all just a few months after having surgery. That feeling is part of why I write these aggressively long posts — to reflect and remember how the music I listened to in 2025 made me feel, not just what it was.

What music discovery means to me

I listened to a lot of music in 2025 because it was familiar and let me focus on my work (my top “songs” of 2025 according to SoundCloud bear that out) — some music will always be functional for me. But I also make a concerted effort to discover new music, or at least new-to-me music Thanks to all my friends who are patient with me when I discover an artist that has been popular for years in different scenes that I am just oblivious to… .

For years now I’ve been listening to less music on Spotify and instead choosing other platforms (SoundCloud, YouTube) and mostly purchasing music on Bandcamp (and occasionally Beatport and the iTunes store when needs must). I wrote about my tactics this time last year: My evolving music discovery pipeline.

As a teen ripping CDs from the library, trolling music blogs for free MP3s and mixtapes, discovering Livejournal groups trading full artist discographies, writing down song lyrics that I heard on the local college radio station or in the store I worked at, I felt a deep curiosity and joy associated with music discovery. And I don’t want to lose that as time goes by!

Like Hanif Abdurraqib points out in A Year of Listening Beyond the Algorithm:

I have kept up my eager pursuit of new music all these years later, because I refuse to believe that the hope of brain-shifting listening experiences must be abandoned with childhood.

I feel the same way — I want to seek out those experiences, to feel something when I listen to music, I want to be challenged and hear new sounds and new tempos and new dynamics.

And it’s working — my music taste has changed since I was 15. There are consistent themes in what I listen to (mad harmonies and a moment where the instruments drop out will always get me) and how I listen to music.

Discovery takes work — it’s not comfortable to always listen to something new. As such, I do my discovery in waves. I’m not always working, but I take steps to seek out new music, genres, producers, record labels, and follow my curiosity.

Discovery for me doesn’t often happen algorithmically. I do covet an excellent autoplay recommendation experience (when SoundCloud’s recommendations hit, they hit hard), but listening to personalized playlists quickly started to feel like a chore with little payoff; a weekly task to be checked off:

√ It’s Monday, so it’s time to listen to Discover Weekly before it goes away.

√ It’s Friday, listen to Release Radar to find out what was released today.

Most of the music in Discover Weekly didn’t stick with me, and Release Radar was reduced to single tracks instead of full albums, often trickling out an entire album’s worth of tracks over weeks of playlists. The more I listened, the more I got in the habit of clicking through on each new track to see if there was secretly a full album of music that I was missing out on.

Nowadays, I’ve abandoned those algorithm-driven playlists.

Instead, I focus more on the personal connections — channels across various social chats where people share music that we love (or that made us feel something) with each other — as well as exploring the track lists of DJ sets on YouTube and SoundCloud, favoriting new releases and collaborations with other producers, and reading posts from music bloggers or others, sharing music that simply could not stay unheralded.

Some ways I was steered toward or discovered new music in 2025 was through podcasts, music blogs, or more organic discoveries and recommendations.

The role of podcasts

Podcasts had an unexpectedly strong role in my music listening in 2025. Unexpected largely because I don’t listen to podcasts very often, but I did discover a new podcast in 2025 — Life of the Record — so the novelty probably helped.

In 2022, I also had a noticeable spike in listening behavior due to a podcast — I listened to a lot of Frightened Rabbit immediately after listening to a podcast about the band and the death of the frontman.

As I mentioned in my favorite songs section, I listened to a lot of Sam Cooke in 2025 because of the Hit Parade episodes about Sam Cooke, Don’t Know Much About History. I’d heard some songs by him, but I didn’t realize just how deep his catalog was of good songs — (What A) Wonderful World is not the Louis Armstrong song, but instead a cute ditty about how much being loved can change your outlook on life. At least that’s my read on it!

Another great set of Hit Parade episodes (that didn’t influence my listening much) was Only Girl in the World Edition, about Rihanna. Powerhouse superstar showed up, made hella albums, and is now a retail mogul. Excellent arc, no notes.

Discovering the podcast Life of the Record also affected my listening, partially because the first episode I listened to was about the Stars album Set Yourself on Fire, which was maybe the first Stars album I heard? Revisiting that, combined with the memory of how unexpectedly raucous they were live for such a delicate sounding (at times) band, turned me back onto their music for a time.

Meanwhile, the podcast also covered a favorite Manchester Orchestra album of mine, Mean Everything to Nothing, which I’d already listened to a few times in 2025 while in a moody reminiscence a la Frightened Rabbit. Inspired by the podcast, I gave it a few more listens, and listened to the albums before and after it for good measure.

End-of-year music wrap-ups

Other people posting their favorite music also influenced my music into the next year. Every year, Sean from Said the Gramophone puts out a top 100 songs list of the year. I love this for multiple reasons:

Music blogs have had a huge impact on my music taste and I love anything that keeps music blogging alive.
Songs are easier to listen to than full albums when processing for discovery.
The list spans genres and reminds me of how much I like songwriters and indie rock and helps me key in to new genres I might like.

Why do I mention this? The only reason I listened to Mk.gee and Waxahatchee at all in 2025 because of that list. And I ended up liking both of their albums quite a bit and listening a lot.

Mk.gee’s album Two Star & the Dream Police ended up my most-listened-to album of the year — and all of my listening happened in January. Sorry Mk.gee, I guess in the summer I forget that I like indie rock.

I haven’t listened to much Waxahatchee since seeing them live (opening for Rilo Kiley) in October 2025, but I spent a lot of early 2025 winter listening to their album Tigers Blood in full, and they ended up my fourth-most-listened-to artist of the year.

Joshua Idehen was a new discovery in 2025, also thanks to the 2024 list.

I treasure the Said the Gramophone post from Sean because my conduits for indie rock discovery are thin on the ground now that I don’t listen to college radio (or DJ in it during the tail end of the heyday). Mostly I’m relying on recommendations from friends.

DJ sets have a unique way in electronic music of introducing you to similar artists and producers, but for indie rock artists, you have to dig deeper to find the occasional collaborators, or go see the artists live and make sure to catch the openers, or similar. It’s a lower-traffic and more active discovery mechanism to follow those discovery threads through without resorting to algorithms.

Organic discoveries

As I mentioned earlier, I discovered 700 “new” artists in 2025, of which several were new favorites. But how did I discover them? Most of them, I discovered organically…

The Beaches I discovered ( again I technically discovered them first in 2017, listening to them 7 times that year, but I listened to them 30 times in 2025 so it feels like a proper rediscovery. ) after a friend at work recommended them, then they released an excellent Tiny Desk Concert.
GLOCKTA released an EP on a record label, ec2a, that I’d previously purchased music from. I took a chance on an EP from them, listening in the background, and boy did it pay off.
IN PARALLEL I discovered purely because they booked a show in San Francisco. I keep track of shows happening in the city (so I can go to them, of course) and based on the promoter and the venue I realized that I’d probably like them, so I looked up a set and it was great.
Dr Dubplate I discovered thanks to the Boiler Room DJ set where he played b2b with Y U QT, favorite producers of mine.
Faster Horses I also discovered thanks to the ~ network of producers ~ that I follow on Bandcamp and elsewise, hearing his tracks played out in some DJ sets I enjoyed and eventually tracking him down on Bandcamp.

These sorts of organic discoveries really stuck with me, because making the discovery took some effort (and a little risk), and it was rewarding.

Why I talk about music discovery with Spotify Wrapped

Maybe I talk about this effortful music discovery every year, but it’s so important to me. Music is a part of culture, and culture is something that we share as a community — it’s something that we make and participate in, not something that happens to us. It’s important to me that I engage with it directly, instead of passively.

Liz Pelly’s outstanding book Mood Machine makes it clear that Spotify has chosen to focus on a specific target consumer market — the “lean-back” listener. As Pelly points out on page 32 of Mood Machine:

In the streaming era, the industry identified a new type of target consumer: the lean-back listener, who was less concerned with seeking out artists and albums, and was happy to simply double-click on a playlist for focusing, working out, or winding down.

Spotify chose to focus primarily on those target consumers, and as part of that move, set up internal structures that built playlists using vast troves of collected and computed metadata.

Instead of building an application that rewarded active listening and focused on adventurous, educational, or thoughtful curation (what might build more of a music culture or community), Spotify chose to “follow the data” of what users of the service were already listening to, and emphasize this “lean-back” listening behavior:

there was nothing inherently neutral about setting up a system that wrapped up all of a song’s worth in its replay value. To do so is to suggest that a song’s potential to ignite mass enthusiasm, and thus mass streams—or its capacity as background fodder, streaming endlessly, unnoticed—should determine its worth.

What Spotify chooses, quite literally, to reward, is the more passive user behaviors like “frequently replayed” or “infrequently skipped”. Spotify has built their business around optimizing for metrics like this, and as such, has shifted (or at least focused) the attention of the music industry accordingly. On page 44 of Mood Machine, Pelly continues:

By championing the lean-back listener, Spotify helped popularize a resurgence of interest in “functional music,” the industry’s current preferred way to describe music for sleeping, studying, chilling, focusing, etcetera. Today, functional music is a defining phenomenon of the streaming era, especially mood music for wellness and productivity, which self-improvement culture increasingly renders the same.

It’s ordinary and typical to listen to music in the background of tasks. I also listen to music this way—I focus better when I’m listening to music, and often listen to music while performing other tasks. I have music on in the background while writing these very words (a track by tennyson). Even for me, “listening to music” is rarely a primary task.

Listening to music in the background, however, doesn’t inherently require “lean-back” listening behavior. I still remember the grooves and the patterns of the Explosions in the Sky album The Earth Is Not A Cold Dead Place, ingrained in my memory after countless repeat background listens in college.

These days, I intentionally try to reward “sit up” listening for myself, and pay attention to the music that sparks something in me (especially when I’m engaged in more mindless activities like commuting to work).

While I still engage in mindless background music listening, I try to intentionally curate the background by listening to DJ sets or live music sets that have been curated instead Like HAAi b2b Romy at the Lot Radio 01-16-2026! and which might introduce me to some new-to-me music. Perhaps this week I’ll listen to some newer Explosions in the Sky or other post-rock albums.

The rating practice that helps lead to music acquisition (if on a streaming site) or frequency of listens (if I already own the song) is my way of recognizing and rewarding the “sit up” listening.

If I’m listening to new music that I just bought or an autoplay recommendation deep in SoundCloud, I take action if a track makes me sit up and take notice. I’ll take a moment to at least heart the track, add a rating, and possibly go further to explore a discography, listen to a few other tracks, or boost the track to others.

I don’t mention this to espouse my personal music listening habits as a virtuous paragon, but rather to showcase an alternate method of engagement with music and streaming services, that goes beyond the default of a Daily Mix in a streaming service and how I have found a way to carve out my own meaning with the media and services I engage in day-to-day.

What’s to come in 2026?

More of the same discovery practices, I expect, with a bit more effort toward indie rock and pop music. The Popjustice folks have some great recommendations, but I rarely encounter them in my usual channels. Indie rock, as I’ve mentioned throughout this post, is not as easy to organically discover (when compared to tracks in an electronic music DJ set).

I think some college radio listening might be in my future to help with discovering more non-electronic music, as well as soliciting playlist collaborations with friends. I don’t want to be so late to the next MUNA, Chappell Roan, or Caroline Polachek — an earlier discovery means more time to enjoy!

Having finished reading Mood Machine in 2025, I have a lot of books on similar or adjacent themes that I’m hoping to get to this year. I’ve built them into a syllabus of sorts, and I’d love recommendations.

capitalism, extraction, and surveillance	tech, culture, and capitalism	culture, art, and meaning
Derivative Media: How Wall Street Devours Culture by Andrew deWaard	Mood Machine by Liz Pelly	What Art Does by Brian Eno
Immediacy, or The Style of Too Late Capitalism by Anna Kornbluh	Computing Taste: Algorithms and the Makers of Music Recommendation by Nick Seaver	Good Music: What It Is and Who Gets to Decide by John J. Sheinbaum
The Age of Extraction: How Tech Platforms Conquered the Economy and Threaten Our Future Prosperity by Tim Wu	Chokepoint Capitalism by Cory Doctorow and Rebecca Giblin	They Can’t Kill Us Until They Kill Us by Hanif Abdurraqib
Surveillance Valley by Yasha Levine	The Uncanny Muse: Music, Art, and Machines from Automata to AI by David Hajdu	Uproot: Travels in 21st-Century Music and Digital Culture by Jace Clayton
Technofeudalism by Yanis Varoufakis	You Have Not Yet Heard Your Favourite Song by Glenn McDonald	Ways of Hearing by Damon Krukowski
The Age of Surveillance Capitalism by Shoshana Zuboff	Categorizing Sound: Genre and Twentieth-Century Popular Music by David Brackett	Reclaiming Art in the Age of Artifice by J.F. Martel

Look forward to more from me on related topics this year…

This year, a calculation error that I made in my math really revealed itself — I thought I had listened to 276,606 minutes of music. Which sounds plausible until you remember the most well-known song from RENT, and that there are 525,600 minutes in a year. I listen to a lot of music, but 12+ hours of music per day is wildly inaccurate. The calculation I perform does several things: First, retrieve my listening data, then look up each track that I listened to that year and append the track_length from my iTunes library, if it exists, to the event. Separately, I calculate an estimated track_length based on the average track_length of all songs I listened to in the year. For each track_name, coalesce the actual track_length and the estimated track_length to make sure each track has a length listed. Then count the track_names and pass through the track length, multiply the count by track_name for each track_length and sum it, then transform length into minutes. I had an error where I split my count by both track_name and track_length, which somehow caused a track I listened to 5 times to show up as 275 listens. Lessons to always gut check my data analysis results, and thank you to Scott online for identifying the mistake! ↩︎
TIL that St. Cecilia is the patron saint of music. ↩︎

Developing an AI strategy for documentation

Thu, 20 Nov 2025 10:44:05 +0000

If you find yourself worried about being left behind because you don’t know what to do about AI and technical writing, this blog post is for you.

Lots of people are using AI. As Anil Dash puts it in his blog post I know you don’t want them to want AI, but…:

using AI tools is an incredibly mainstream experience now.

Even if you’re not personally using AI, many readers of the documentation you write likely are, which means your documentation needs to meet them where they are. You need an AI strategy for your documentation.

Why you might need an AI strategy

Information discovery is changing. When you’re looking for answers to a question you have, you probably do a search on Google. It’s been years since Google launched, so it feels like second nature.

Before search engines existed, people looked up information in other ways — asking librarians, reading encyclopedias, visiting a trusted website (shoutout Encyclopedia Britannica). After search engines launched, people started to look at the search engine results first (thanks, Ask Jeeves). And if the results were useful and relevant (and they often were), people started to trust the results presented by search engines and rely on them when seeking information.

Someone looking for information about your software product can review results containing your documentation, but also third-party resources, and YouTube videos:

However, with the launch and popularity of ChatGPT, Claude, and other AI-based tools, habits are shifting again for many people. People are experimenting with different ways to seek information. One of those ways is using AI to learn about things:

Over the past few months, I’ve been talking with folks about how they find information and learn about a product. ChatGPT and Claude and Gemini (or at least Google’s “AI overview”) are consistently mentioned as one of the main methods people use. Even if the output isn’t that reliable, visiting an AI tool is still a key part of the information-finding flow.

AI tools are used for all sorts of information-finding flows:

Answering basic questions about how to accomplish a task.
Decipher the potential root cause of an error message and provide next steps for resolution.
Provide solutions to intricate application design questions.
Suggest syntax for code, a formula, or a SQL statement.

You need an AI strategy for your documentation to enable customers using AI to be more successful when using your product to accomplish their goals.

Define your AI strategy

The components of an effective AI strategy include the following:

Develop: Partner with teams building AI into your product.
Discover: Evaluate and improve your content and content strategy.
Measure: Measure how people use AI to interact with your documentation.
Adapt: Explore use cases for AI in your documentation practice.

Partner with teams building AI into your product

If you partner with teams building AI functionality into your product, you can enrich those product-relevant AI tools with documentation context.

Two common modalities for in-product AI tools are chatbots and agents:

AI chatbots explaining tasks in the product.
AI agents performing tasks in the product Oftentimes, agents have some sort of explanatory component. For example, an AI agent that rewrites documentation markup to fix a syntax error is functioning as an agent, but it might perform tasks as part of a conversational interface like a chatbot. .

For both chatbots and agents, documentation is a critical component to ensuring the success of the functionality. Why? Because documentation provides the authoritative source of instructions for performing tasks successfully with the product.

If an AI agent is provided with instructions for performing tasks, it can be more effective in completing those tasks, which makes the person trying to use the AI agent more efficient and successful at their goal.

For an AI chatbot, an ideal experience can look like what Casey Smith describes in 10x Impact: Inside Payabli’s Documentation Revolution:

Customers can ask questions like “build me a config for this service,” and the chat generates working configurations from our documentation.

The process of providing in-app documentation is a lot less complex when one interface can provide on-demand assistance. Instead of a clunky experience where you’re configuring elaborate mappings between product pages and documentation pages, or attempting to make webpage-sized content coherent in a 100px square popover.

If your customers have an existing pattern of completing tasks that your company is already planning to add AI into, make sure you’re involved so that you can add documentation context.

Hot take

You can invest in building a chatbot with a tool like Fin, Inkeep, or Kapa. But I don’t recommend placing the chatbot on the documentation site.

When you build your own chatbot and provide it on your docs, you create uncertainty for your customers and readers:

Is the response going to contain accurate and trustworthy information?
Is this chatbot going to be more reliable than ChatGPT or Claude?

Documentation is perceived as an authoritative source of information. If the official documentation provides a way to get inaccurate or misleading information, readers might lose trust in the documentation itself, and ask: Are AI chatbots docs? (No.)

When customers get stuck with your product, they often have to leave to get help. The advantage of an AI chatbot is that it’s a simple way to provide (qualified, yet potentially unreliable) assistance with using the product inside the product. People expect a chatbot within the product to have context of the authoritative how-to documentation because it exists within the product itself.

Instead of adding yet another interface or entry point for customers to learn, discover, and interact with, focus on providing assistance with the product inside the product.

As part of your AI strategy, identify if there is a team building AI functionality into your product, and then partner with them. If there isn’t, partner with someone in engineering to explore options or opportunities for adding an in-app documentation chatbot that can provide that on-demand assistance.

Improve your content and content strategy

Writing high-quality content for humans also helps AI-based tools, but there are still some accommodations that you can make in your content to help AI tools make better sense of it:

For other guidance on writing for LLMs (and humans too), see the following resources:

The blog posts listed in Optimizing docs for LLMs in the November 2025 Write the Docs newsletter.
Writing documentation for AI: best practices by Kapa.ai, a tool that provides retrieval-augmented generation (RAG)-based chatbots.
Optimizing content for Fin by Intercom, who provide a RAG-based chatbot called Fin.

Expose your content to LLMs

LLMs perform markedly better at answering questions about how your product might work when they have access to the context available on your documentation site.

To help improve the likelihood of high-quality answers from LLMs, implement the following practices:

Set up an LLMs.txt file, essentially a sitemap that points to the raw markdown files of your content, which is easier for LLMs to process.
Expose a model context protocol (MCP) server for your documentation site to enable AI agents to search and fetch results from your documentation, or even perform interactive tasks in your API documentation.
- For more about what MCP is and how it might help, see The Model Context Protocol (MCP) and its impact on technical documentation from Cherryleaf.
- For even more details, check out the podcast episode MCP servers and the role tech writers can play in shaping AI capabilities and outcomes with Tom Johnson, Fabrizio Ferri Beneditti, and Anandi Knuppel.

Provide decision support and user-centric content

Customers use your product to accomplish goals they have, like “keep track of my work”. Customers aren’t thinking “today I am going to use a Trello board and cards”.

If you write feature-focused documentation, you need to make sure your content ties together the features in coherent, user-centric content that addresses customer goals.

For example:

Feature-focused	User-centric
Create a card on a Trello board	Track tasks on a Trello board

Or you can split the difference: Create cards to track tasks on a Trello board.

Your customers might already be asking for this type of content (or silently wishing for it), but the LLM tools also need user-centric content to help answer questions about your product more accurately.

When people look for help, a product feature might be the answer, but it’s not the question. People ask chatbots questions, so your documentation must address the question too. Don’t write FAQs (really, don’t). Instead, write user-centric content that addresses both the question and the answer!

If your content only includes information about creating cards, the LLM might make something up because the context about the user goal (tracking tasks) is missing from the documentation. A human might do a decent job at guessing based on prior mental models of task-tracking software, but an LLM needs more help to make that association Trello isn’t a great example because products that are available directly to consumers have somewhat less of a risk with this because there is a substantial amount of third-party content also being created that might fill that gap for your documentation. .

To alleviate the risk of hallucination, write user-centric content that includes decision support to help address questions like “How can I track tasks in Trello?” without literally writing content titled “How can I track tasks in Trello?”.

Write precise and clear content

When LLMs process content from a webpage, the content is split into pieces. For a RAG-based chatbot, this processing is referred to as chunking, and LLMs like ChatGPT and Claude can also chunk content to make retrieving context from linked sources more efficient.

Due to this information processing method, you might want to invest more effort on writing precisely (and possibly also update your style guide!). As quoted in the Write the Docs November 2025 newsletter on Optimizing docs for LLMs:

according to a recent Intercom article by Beth-Ann Sher (Optimizing content for Fin) writing “as if you’re doing a radio interview and you don’t want to be quoted out of context” can be more effective than FAQs.

Given that guidance, it’s crucial to avoid language shortcuts like the following:

Unclear antecedents
Excessive use of relative pronouns
Positional or relational language like “below” or “earlier”

Instead, make references explicit. Refer to specific steps by number, titles of tables, or restate the subject instead of using a relative pronoun. Your writing might get more verbose, but it will also get a lot more clear.

For example, given a section of content like:

Before you eat dinner, set up a table for the cheese platter. This helps ensure that people can access the cheese at any time. It should include the following:

Cheddar

Emmentaler

Gouda

For guidance selecting an appropriately sized platter, refer to the steps above.

You might instead rewrite the section of content to read differently:

Before you eat dinner, set up a table for the cheese platter to ensure that anyone can access the cheese at any time. The cheese table should include the following types of cheese:

Cheddar

Emmentaler

Gouda

For guidance selecting an appropriately sized cheese platter, refer to step 2 of this hosting guide, “Select your serving dishes”.

I’ve exaggerated these examples, but as writers, we often take language shortcuts like using relative pronouns or unclear antecedents. By taking the long way and making these references explicit, not only can LLMs more easily process the content, but so can people that process information differently.

For more details about these writing improvements, I recommend the guidance to Be precise in the Splunk Style Guide (which, for the record, predates LLMs).

Measure how people use AI to interact with your documentation

An important component of your AI strategy is understanding how many people are using AI to learn more about your product and interact with your documentation. For the most part, you can measure it!

In addition to SEO (search engine optimization), web marketing has started to refer to AEO (answer engine optimization) and GEO (generative engine optimization) as new acronyms to care about for measuring and optimizing content for LLMs and AI tools.

The tactics for SEO vs AEO or GEO are similar, but measuring AEO and GEO requires a bit more effort than measuring SEO.

As part of measuring AEO and GEO, I suggest expanding your web analytics monitoring to measure the following for your documentation:

Track referrer traffic originating from LLM chatbots, like chatgpt.com or claude.ai.

Don’t consider referrer traffic a complete representation, however, because tracking only referrers is inherently incomplete due to limitations with how different sites and browsers handle referrer traffic.
Identify AI-attributable traffic in the user agent strings, such as traffic from AI crawling bots or attributable to an MCP server enabled for your documentation.

For a list of known user agents, see Major AI Bot User-Agents on llmsCentral, or AI user-agents, bots, and crawlers to watch from Momentic Marketing.

If you have server-level website monitoring, you can also track more data in request headers, looking for the Signature-Agent field in traffic for your site to identify traffic from AI agents. Read more in Simon Willison’s post ChatGPT agent’s user-agent.

Evaluate the performance of AI tools when answering questions

Evaluating the performance of AI tools for answering questions is also something you can do as part of an AI strategy for documentation This level of evaluation is extremely time-consuming and resource-intensive, so if you’re only planning on implementing a retrieval-augmented-generation (RAG)-based documentation chatbot, evaluating LLM performance at this scale is likely excessive. . However, this is another case where I’d recommend partnering with any teams working internally on AI functionality to ensure that you aren’t duplicating efforts.

When evaluating the performance of AI tools, you can pay for large-scale QA through tools like Profound or Amplitude, or perform and track evaluation suites that you develop with tools like Gauge or Langfuse.

If you perform manual QA, devise a set of high-value questions to evaluate, choosing from the questions that customers most frequently ask support. If that data isn’t available to you, consider the following sources:

Common information that customers search for, guided by particularly long queries typed into search.
High-stakes questions that are important to get right about how your product works.
Comments that people leave in page quality feedback responses for the documentation site.

After you identify the set of questions, define a high quality “ground truth” answer for each question to measure the performance of the LLM-based tools against.

After you have the questions and the answers, ask the questions, record the responses, and score the accuracy of the results against the ground truth answer.

Performing evaluations like this over time helps you monitor the quality of results that customers encounter when looking for information.

If you plan to make changes to your content to help LLMs, perform these evaluations before making those changes to develop a wobbly baseline Performance and quality of LLM-based tools is somewhat inconsistent due to the changes in model performance, system prompts in use, and other factors. Therefore, a wobbly baseline is all that I think you can expect. for your product and documentation performance.

Then, after making changes, repeat the evaluation after some time has passed. I recommend waiting at least a week, or even up to a month, to test changes in external systems. With any luck, the external AI tools will be more successful at providing more reliable or higher-quality responses for the questions being asked when compared to your wobbly baseline.

Consider testing your documentation directly

If you specifically want to evaluate how well your documentation is optimized for an LLM, you can try asking the AI tool directly, as Casey Smith suggests in AI chat as user research when your reader is a bot:

Running content through an AI chat to ask what it thinks is one of the greatest uses for AI as a technical writer. I’m thinking of it as user research, only my user is an LLM with a chat interface. With the adoption of AI chatbots, it makes sense to include them in our content testing and review plans, right?

The important distinction here is that you’re not replacing user testing with a chatbot, but instead testing the success of a different kind of documentation user — the LLM-based chatbot. This sort of testing is one way to evaluate how effectively an LLM-based tool can retrieve relevant content from specific pages of your documentation when directly prompted to do so.

Explore use cases for AI in your documentation practice

Many teams at software companies are being required to use AI. If you’re not yet in that situation, you might be wondering how long you have until you’re asked about using AI in your work.

If and when your leadership team asks you what your plans are for AI content creation, you want to be prepared. Compile a list of what you might use AI to accomplish in your work.

A list of use cases makes it clear that you’re thinking about AI, and that you’re thinking beyond the most basic consideration for using LLMs in technical documentation:

  ---
config:
  theme: 'base'
  themeVariables:
    primaryColor: '#a32fab9f'
    lineColor: '#a32fab9f'
---
graph LR
  A[AI can generate content] --> B[Your job is to generate content]
  B --> C[AI can do your job]

As Fabrizio Ferri Benedetti breaks down in his excellent post Code wikis are documentation theater as a service, generating content from code is not an effective use case for AI in technical writing, so think beyond that.

You want to be strategic and consider where AI might add value to your documentation process. What are the chores and tasks that you don’t like to perform, or that you haven’t found time to do because it would require too much expertise that you don’t have at the moment?

For each use case on your list, consider how you can test the output and how you will keep a human in the loop to ensure quality output. How do you know that the AI is performing better than a person might? Can you measure the effect of using the tool on your productivity?

Some example use cases that might fit on your list of AI use cases for technical documentation:

Extend your static site generator to include a custom theme for admonitions using CSS written by an AI assistant.
Draft a template for new documentation pages to simplify contributions.
Develop style guide linting rules in Vale based on your current style guide.
Use an AI-based code reviewer like CodeRabbit to review PRs for style and valid syntax.
Write alt text for images that humans can review for usefulness and accuracy.
Convert a screenshot of a diagram into a mermaid diagram.
Generate sample data for an interactive tutorial.
Write a script to auto-generate reference content from code.
Analyze the metadata of your documentation pages to build a knowledge graph.

The Write the Docs newsletter from July 2025 includes some additional use cases. See How documentarians use AI (or LLMs).

Leadership might not know the intricacies of your job, but you can demonstrate both your ~ growth mindset ~ and your expertise by learning about the tools available to you and proposing ways to use them Spoilers: You don’t have to actually use them! .

Implement your AI strategy

AI is changing the way people discover and learn. As a result, to grow and learn as a technical writer, you need to know how to deliver content to those people.

Developing an effective AI strategy for your documentation workflow doesn’t mean adopting everything I’ve described here. Pick and choose what makes sense for your organization, personal bandwidth, or team maturity.

Partner with teams building AI agents and chatbots in your product
Make your content easily available to AI chatbots and agents
Deliver conceptual decision support content
Write precise and clear content
Measure AI-driven web traffic
Evaluate LLM performance for specific product questions
Explore AI use cases for your own workflows

No matter how tempting it might be, try not to ignore the existence of AI entirely. Defining your own AI strategy is both a proactive and defensive endeavor — learn about a new technology, and defend your expertise with that knowledge.

Save a horse, use a content strategy

Thu, 13 Nov 2025 18:30:13 +0000

You can lead a horse to water, but you can’t make it drink. In this metaphor, technical documentation is the water.

You can spend a lot of time making the pond look nice and accessible. Provide clear, clean water, some lily pads, a water feature to make it easier to drink. You invest in the structure of the pond to make sure the water doesn’t get stagnant, and even add a nice sign to indicate “fresh water here”.

Credit: Simon Carey / Horse Pond / CC BY-SA 2.0

But if the route to the pond is treacherous — difficult to follow, unclear, full of pitfalls — then the horses won’t visit your beautifully curated pond. You won’t be able to lead the horse to the pond with the fresh, high-quality water. Instead, the horses will drink out of a muddy puddle — because the puddle is right in front of them where they were going already.

I’m stretching this metaphor to its limits to underscore the importance of having a content strategy. It’s not enough to write effective, accurate, and useful content. You also need to identify how people discover your content — invest in documentation discovery and evangelism.

If people don’t know how to get to your content, if it doesn’t appear in their pre-existing paths, they won’t discover it. And if they don’t know what’s in it or what the quality of your content is, they won’t use it.

Consider the context

When you write documentation, you must consider the context in which it exists:

How do customers learn about your product?
How do they discover helpful content?
What do they do when they have a question?
Where do they go for help when they get stuck?

If your documentation isn’t available and accessible where your customers need it, it almost doesn’t matter what you write because it won’t get used.

Instead, customers will turn to alternatives:

Google search AI overview
Chat support with live humans
LLM-based chat interactions like ChatGPT or Claude
Training from third-party consultants
Blog posts published on your company website
Videos created by external developers

These content types can be useful, but many of them are expensive to provide (and access) and are frequently not well maintained. Resources that are only available to some folks or are outdated are not as useful, reliable, or authoritative as official documentation. And if customers are using these content types instead of rather than in addition to the official documentation, you have a problem.

Improve access and awareness

So what can you do?

Ask people what they do when they get stuck. If they don’t mention the documentation, ask them why — identify the hurdles hindering access to the pond.
Make the documentation easy to get to. Pave the way to the beautiful pond you’ve curated by adding entry points like links from the website and the product and improving SEO.
Evangelize the documentation. Tell people about it, talk about the great resources available internally to the folks that interact directly with customers.
Shift priorities to focus on enhancing discovery instead of writing feature-driven content. Accept that the pond can grow some algae as long as the water remains potable.

You might discover when you talk to customers and internal stakeholders that the documentation you thought was excellent isn’t actually fitting the need — maybe the pond is too shallow to provide effective hydration to the horses, or the shoreline is difficult to access, or only some horses can find their way to the pond.

Getting feedback can help you prioritize your work accordingly to help ensure that your documentation gets used. If your technical content isn’t available where your users look for it, your content won’t get used — no matter how good it is.

Humans are creatures of convenience. Build your beautiful pond, but don’t forget to make it easy to access.

Credit: Sunny Ripert / Chevaux au bord de l'eau / CC BY-SA 2.0

Music metadata: What it is and why it matters

Tue, 21 Oct 2025 22:49:22 +0000

A few weeks ago, I gave a lightning talk about music metadata. It’s one of my favorite niche topics to research, so talk covers a lot of what I know about music metadata.

Welcome to my talk about music metadata.

In this talk, I’ll cover what music metadata is, how it’s made (because it’s not discovered!) as well as what it’s used for — and why it matters. If you want to learn more, I’ll also talk about how you can interact with it.

Music metadata is crucial to the functioning of digital streaming platforms that we use every day.

Music metadata can be roughly categorized into three main types, as outlined by a 2011 patent from the Echo Nest, Automatically acquiring acoustic information about music:

Acoustic metadata: the mathematical representation of the sound of a track
Cultural metadata: text that describes the content
Explicit metadata: the factual information about the track

Acoustic metadata is the “numerical or mathematical representation of the sound of a track”.

Some examples of acoustic metadata are things like duration. If you’ve ever explored the Spotify API, you might be familiar with other metadata fields like danceability, tempo, speechiness, or valence.

These example values are for SZA’s track Kill Bill (feat. Doja Cat):

Cultural metadata is “text-based information describing listener’s reactions to a track or song”.

Some examples of cultural metadata are things like genre tags that you might see on a SoundCloud track, names of playlists that the track is added to, tags added to the track in a service like Last.fm, hashtags on TikToks that sample audio from the track, even YouTube comments. It can also include things like album reviews, blog posts and comments about an artist or track.

I’d guess that the popular music blog aggregator the Hype Machine has probably been scraped to help build out cultural metadata examples. The examples on this slide are for Carly Rae Jepsen.

Explicit metadata is “factual or explicit information relating to music” — the things that you’re probably most familiar with and that you might immediately think of when you think of music metadata.

For example, the track name (Diamonds), artist name (Rihanna), including any featured artists as relevant, the release date (1 January 2012), as well as the C-line for copyright purposes (© 2012 The Island Def Jam Music Group) or the P-line to denote who owns publishing. The Sky Ferreira track “Nobody Asked Me (If I Was Okay) has a P-line of ℗ 2013 Capitol Records, LLC.

All of these examples are explicit metadata values that might be associated with a track.

You might be wondering about user behavior data? What about all my listening data that shows up in Spotify Wrapped or whatever?

That isn’t metadata about the music itself, but it is still relevant. And much like music metadata, the user behavior data that these platforms are collecting and defining are similarly opaque yet specific.

The combination of user behavior data and music metadata is what makes the other type of data valuable. For digital streaming platforms like Spotify, Apple Music, Tidal, Deezer, SoundCloud, and Pandora, having user behavior data to know that people are listening to music on the platform is useful.

But user behavior data is relevant for being able to develop categorization, clustering, and recommendation algorithms using the music metadata, which is extremely valuable. With the combination, you can calculate things like:

Track duration before skipping
Geographic region of listeners
% of mainstream music listened to
Track popularity among platform listeners
Volume of repeat listens
Number of playlist placements
Likes and dislikes during radio playback
User gestures like swipe
Playback devices used
Total monthly listeners
Total minutes listened to an artist

You might be wondering where music metadata comes from — and how it is made.

When I did research about this several years ago, I discovered that for the most part, music metadata is:

defined by algorithms written by trained musicologists, amateur musicians, or even ordinary data scientists without musical training or expertise.

Depending on the type of music metadata, there are different ways that the metadata might get made.

For acoustic metadata, manual annotation is the most relevant. If you’ve ever heard of the Music Genome Project from Pandora, that was expert musicologists, about 450 of them, at least in the beginning of Pandora, defining all sorts of metadata fields for music.

Nowadays, the manual annotation is more likely to be everyday data labelers performing something like human-in-the-loop quality assurance for tracks by emerging artists or in newly added regions, after being processed by algorithms and machine learning models.

After you have a manually annotated “ground truth” dataset, you can build algorithms and machine learning models to perform the labeling programmatically, using a split version of the manually annotated dataset as a way to both train models and test the outputs.

The algorithms and machine learning models are performing tasks like spectrogram analysis or audio fingerprint creation (part of how Shazam works, and also copyright detection like YouTube’s content ID), as well as analysis to derive acoustic metadata more generally.

For example, Spotify filed a patent for Automatic prediction of acoustic attributes from an audio signal, wherein they outline the attribution of more “subjective” acoustic metadata from a song:

While music is composed of objective properties (e.g., tempo, onsets, durations, pitches, instruments), its audio recording carries inherent information that induces emotional responses from humans, which are typically problematic to quantify. Subjective and lacking a base unit, like “seconds” for duration, or “hertz” for pitch, these responses may nevertheless be fairly consistent across the spectrum of listeners, and therefore can be considered intrinsic song attributes. Subjective attributes include for instance the measure of intensity or energy of a song, the valence or musical positiveness conveyed by the song, and the suitability of a song for a particular activity (e.g., dancing). Other song attributes, however, are reasonably objective but hardly detectable from the structure of the music, its score representation or its transcription. These include, for example, whether a song: was recorded live; was exclusively recorded with acoustic instruments; is exclusively instrumental; and the vocals are spoken words.

The patent, being a patent, goes on to detail the methods used to codify those subjective acoustic metadata values for a song into numeric terms, including the methods of collecting labeled audio content and the methods for predicting new values for unlabeled audio content.

Cultural metadata is also going to be manually annotated. The people performing the manual annotations for cultural metadata are oftentimes going to be slightly less expert data labelers. These days they’re usually performing quality assurance of machine learning categorization of tracks.

A lot of cultural metadata is also derived from what is published online about a track or an artist. The metadata comes from what people are saying in music blogs, forum discussions, public social media groups, or in the genre tags added to sites like Last.fm, even the hashtags used on a TikTok video or Instagram reel that uses the audio might be considered.

Anything that is on the public Internet can be scraped, cleaned up with a tool like Beautiful Soup, and then processed. Cultural metadata can be derived from the specific words or phrases used to describe a track, or even from the sentiment of language used—classic natural language processing tasks.

You are also contributing to cultural metadata. Surprise! All those playlists that you create on your favorite digital streaming platform, the names and descriptions of those are being collected and associated with the tracks in those playlists, and used to inform the cultural metadata for those tracks and artists.

The folks being paid to curate playlists for platforms like Spotify or Apple Music are also generating cultural metadata, categorizing tracks by mood, vibe, or genre based on the playlist they curate.

Other cultural metadata that everyday music listeners contribute includes comments that we leave on tracks on sites like YouTube or SoundCloud.

Explicit metadata, as you might expect, is mostly coming directly from the artists, publishers, and distributors themselves.

And surprise, that means it’s manually annotated. Interns (or better-paid roles) at the record labels perform the data entry to provide the explicit metadata, and content operations and quality assurance teams and the digital streaming platforms themselves review and update the metadata to make sure it’s correct.

The publishers and distributors of tracks also make sure that the explicit metadata is added or included when the track is distributed to streaming services, radio stations, and more.

The manual annotation and review process is important to make sure that the right songs show up in the right places in the streaming platforms, and to help prevent fraud and make sure the right people can get paid.

Some explicit metadata is added programmatically, like the unique IDs assigned by the streaming service (Spotify ID), a MusicBrainz identifier to associate the track with its unique metadata record in that dataset, or other programmatic IDs.

What is music metadata used for? You probably have some guesses. Honestly, music metadata is used for everything you might guess, but also more.

Like I mentioned earlier, there is a good amount of correlation with that user behavior data.

This is from the 2022 Spotify wrapped, where Spotify put together a sort of listening personality that was a little bit like, what if Myers-Briggs, but music?

My assigned personality was The Specialist, featuring the qualities of FNVU: Familiarity, Newness, Variety, and Uniqueness.

To derive a “personality” based on those characteristics, they had to use metadata like song release date, explicit metadata, as well as popularity, and for familiar to you versus unfamiliar to you, how frequently you listen to specific artists.

Those are pretty simple things to compute with a combination of user behavior data and pretty basic metadata. I even calculated it myself in 2022, essentially looking at the following questions:

Does song age affect listen frequency?
Do you listen on shuffle or to albums?
Do you listen to popular music or obscure music?
Do you listen to familiar artists more or less than new artists?

Music metadata is also used for so much machine learning. Machine learning is probably about 90% of what music metadata is used for.

Of this list, I highlighted some interesting use cases that you might guess, but also ones that you might not guess.

One example is playing ads to you according to your user taste profile that Spotify has defined according to your listening (plus metadata) so they can sneak ads in there where the sound and the music of the ad fits in with the music that you’ve been listening to, so that maybe you’re more likely to listen to the ad.

Music metadata and machine learning are also used for more typical tasks, like sequencing tracks, generating playlists, making recommendations, grouping similar artists, or modifying tracks to split them into stems or more simply equalizing audio playback.

Music metadata is also used for marketing. You’re probably familiar with Spotify Wrapped, Apple Music Replay, SoundCloud Playback, and more. But that’s also going to be a combination of that user behavior data and the metadata, the advertising I mentioned.

What is your listening personality? Your music evolution? Your audio day? Your streaming habits? Your musical aura?? The marketing wants you to wonder.

Music metadata is also used for advertising, whether for the example I mentioned earlier where ads are played with similar sounds as the music you’re listening to (helping make the ads more effective), or to recommend ads to you based on your user taste profile defined by the music that you listen to.

And music metadata is also provided back to artists and record labels (although usually only in aggregate metrics rather than raw data). Because they have amassed this enormous amount of user behavior data and the music metadata that they’ve created, digital streaming providers like Spotify can provide metrics on pages like Spotify for Artists.

Artists and their labels can then access metrics like “which playlists are your tracks being added to”, “which tracks are being streamed or saved to user’s libraries”, or “top cities where your listeners are streaming from” and then ostensibly use that data for their own marketing and promotion campaigns. Depending on the terms of various record label contracts, additional metrics might be available for a fee, but the contents of such contracts are not publicly available.

Metadata is also really important for calculating royalties. Different people who contribute to a track are entitled to different shares of the music royalties, depending on where the music is purchased or played.

Publishers (who own the masters) own the rights to reproduce the work and distribute reproductions of the work.
Performing artists (anyone who is credited with having performed on the track) are the named artist and featured artists listed on the track. If you’re not listed on the track, you don’t get paid for being a performer.
Record labels are typically assigned the rights of performing artists in exchange for a record deal and an advance, so they’ll often be allocated the share that would ordinarily go to the performing artist(s).
Songwriters (who wrote the lyrics, or is credited with having done so) are owed a negotiated portion of the publishing royalty.

If you’re not familiar with music royalty calculation, this is one graph that represents one type of royalty that is collected. I think this one is specific to interactive streaming services (which are not all streaming services, just one type) published in Money for Something: Music Licensing in the 21st Century by the Congressional Research Service, updated February 23, 2021.

You can see highlighted in different colors the different types of rates that get paid out and the different places and people that they are paid out to. So, ASCAP and BMI are performing rights organizations in the United States, and they handle payouts to songwriters and music publishers primarily.

SESAC and GMR are performing rights organizations that make it easier to license music (especially internationally), so they handle the payouts from the international streaming services to the songwriters and music publishers, as well as the record labels and featured artists on the tracks.

Everyone gets a different cut according to their role on the track, and the role on the track is defined in that explicit metadata. So if that is wrong, stored somewhere weird, or duplicated, someone is not getting paid.

As you might guess, we’re in this new ~ * magical * ~ AI era, and as such, music metadata is also used to generate music.

Some example prompts that people might use that are going to reference specific acoustic, cultural, and explicit metadata fields could be things like generate a live track, so that’s going to use liveness, featuring trumpets, so that’s going to have to do with that instrumentalness.

Or make it 140 BPM, make it a pop song, make it the country song, but with trap beats… give me 2010s blog rock energy, similar prompts.

The only reason that prompt-based AI generation of music can work is because models have access to the cultural, acoustic, and explicit metadata that can correlate a description of a track with the mathematical representation that is stored digitally.

So, why does music metadata matter?

Music metadata directly influences what you listen to. The valence of track that you listen to (or many tracks) might be higher, indicating that you like happy music, so recommendation algorithms show you more music that is happy. Or similarly, if you listen to a bunch of sad-seeming music, you might get recommended more sad music (which might lead to you being sad for longer).

In addition to being used to recommend similar tracks to you, music metadata can influence track placement on playlists. If a popular playlist is generated by a machine learning pipeline, music metadata like genre (cultural) or bpm (acoustic) affects which tracks are selected for the playlist. Even if a human curator is in-the-loop, metadata characteristics can help narrow down the possible tracks to be included into an easily reviewed list.

Listening to recommendation playlists (like Spotify’s Discover Weekly or SoundCloud’s Weekly Wave) or curated playlists (like Spotify’s RapCaviar) drives music discovery and informs what you listen to, driving your music taste in a specific direction.

The more you listen to streaming services and rely on the playlists and recommendation algorithms, the more your music habits depend on music metadata.

The tracks included on curated playlists that you listen to drive your music discovery and music taste over time, directly influencing what you listen to.

Music metadata can also be fairly biased and inaccurate. I did a small research project last summer to review sped-up songs and identify, on average, how much they were sped up compared to their regular songs. Unfortunately, a lot of the tempo data was just totally inaccurate, which made such an analysis quite difficult. Tempo data is frequently doubled (or halved).

In addition, some acoustic metadata values like valence don’t have an objective baseline, which makes the accuracy of the metric difficult to evaluate. For that metric and others, they are likely defined in relation to something, but that relative definition is opaque. Other metadata values are inherently subjective, like instrumentalness. Even the Spotify API definition for the now-deprecated Get audio features for a track endpoint makes that subjectivity clear:

Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.

Music metadata also directly affects how artists get paid, like I mentioned. If explicit metadata is missing, such as artist data, or if the role is inaccurate, or if there is duplicate or inconsistent data, it gets more difficult to identify and pay the rightsholders.

Metadata is also used to profile you in combination with your user behavior data of a given streaming service. Use cases like clustering listening data into different advertising categories, or specific tasks like personalizing music in the podcast ads that you hear.

If you remember nothing else, remember that the data labelers that are processing and tagging metadata are the tastemakers of the AI era. Data labelers directly control which playlists are being generated and which songs are being added to them, and that shapes what we listen to.

Andrej Karpathy, cofounder of OpenAI pointed out in a post on X that:

People have too inflated sense of what it means to “ask an AI” about something. The AI are language models trained basically by imitation on data from human labelers. Instead of the mysticism of “asking an AI”, think of it more as “asking the average data labeler” on the internet.

Data labelers create music metadata, that metadata is used to inform music streaming, and as such, your experience on a music streaming platform is effectively curated by data labelers.

So what do you do with this knowledge?

One thing you can do is explore the metadata yourself. Play around with it, discover what is cataloged and defined, assess its accuracy, and discover what it can and cannot tell you about music.

You can use a few different methods to retrieve music metadata:

API
CCPA request
Open source datasets
Metadata datasets

Spotify’s API is pretty easy to use with the Spotipy Python library in my experience, and you can get detailed explicit metadata, limited acoustic metadata, and some cultural metadata.

There aren’t that many ways to get acoustic metadata for free. Spotify recently deprecated their most interesting API endpoints and options for retrieving acoustic metadata: Get Track’s Audio Features and Get Track’s Audio Analysis, probably because they were being used to build machine learning models, although it’s interesting that Spotify only felt threatened enough to deprecate the API endpoints in late 2024.

From other Spotify API endpoints like Get Track for, you can get details like:

Track name
Album, album type, total number of tracks, album name, release date
Artists on the track
Track duration, whether the track has explicit lyrics
Unique identifiers like the Spotify ID and external IDs for the track like ISRC or EAN
Popularity of the track

For an endpoint like Get Artist, you can get details like:

Artist name
Total number of artist followers on Spotify
Genres that the artist is associated with
Spotify ID
Popularity of the artist, calculated from the popularity of the artist’s tracks.

Mostly explicit and cultural metadata.

Another API endpoint, for the Last.fm API, only requires an API key for requests (no complicated OAuth flows). You can retrieve explicit metadata and some cultural metadata from Last.fm, but no acoustic metadata.

For example, for an endpoint like track.getInfo, you can get details like:

Track name
Track duration
Total number of track listeners and total play count for the track
Artist name and album name for the track
MusicBrainz Identifiers for the track, artist, and album
Top tags for the track, but what makes a tag “top” is not defined by Last.fm beyond mentioning that they are “ordered by tag count”.

For more details about the tags assigned a given track, you can call the track.getTopTags endpoint and retrieve:

Number of tags for a tag
Tag name (like “hard rock”, “rock”, classic rock” for AC/DC)
URL for the tag (where you can view top artists, tracks, albums, and related tags for the tag)

For an endpoint like artist.getInfo, you can get details like:

Artist name
Total listeners and plays (play count) for the artist
Similar artists
Tags for the artist
Bio details for the artist

If you use Last.fm (like I do), you can also retrieve your own user behavioral data (listening behavior) as recorded by the various systems that you use to record listening activity and send it to Last.fm. I mention that caveat because what counts as a “listen” can vary across systems, which can affect the accuracy of the collected data.

Beyond simple programmatic methods of collecting your own music metadatasets, you can also request additional data. If you’re in the United States and a California resident, or live in a European country subject to GDPR, you can usually make a data request of specific services.

For example, I’ve made a CCPA request in the past to request my Shazam history for every single song that I’ve Shazammed. You get a history of your user behavior in the Shazam app and service generally, but also a CSV-formatted file containing a record of your synced Shazam results — artist, title, and date (although if you allow Shazam to use location services, you might also get the longitude and latitude where you Shazammed the track).

Beyond these methods, you can access an open source dataset or another metadata dataset. The MusicBrainz identifier output in the response of most Last.fm API requests aligns with the MusicBrainz Database, which functions as an publicly available database of music metadata.

Other datasets are those like the Million Song Dataset, primarily created for research purposes, but which contains acoustic, cultural, explicit metadata. Beyond that one, Kaggle and Hugging Face offer hundreds (if not thousands) of other datasets to choose from.

The creation methods of music metadata might be opaque but you can still explore the output.

Some questions I was asked include:

What is your personal favorite piece of musical metadata, and how have you used it?

Tempo is probably the one I’ve spent the most time with, in attempting to analyze the difference in bpm for sped-up songs and the original versions. However, that project ended up being mostly an exercise in data cleaning, so it wasn’t nearly as exciting as I hoped it would be.

Maybe the funniest piece of metadata is valence, because it is sort of just like, “what if we put a number on how happy this song sounds?”, which like, okay sure, that’s a thing you can do.

I mostly care about, like, the accuracy of the metadata and how arbitrary it is to measure some of these qualities that we made into math and made them feel objective, but they are mostly subjective.

How does the emergence of new genres work? Like, sped up is, like, a thing that people talk about in the past couple years, but my understanding is, like, early 2000s, we call it, like, Nitecore, or, like, meme rap is a thing, but it’s only popularized in the past 10 years, and all that.

That’s where the cultural metadata is coming from, where it’s like, maybe someone mentions it on a blog, people start to recognize patterns and themes in the data that they’re scraping and encode it in the cultural metadata that gets built.

There’s also a certain element of, like, Spotify just makes stuff up. Like, Spotify might notice patterns or trends in terms of clusters of artists, and then they’ll decide “this is a micro-genre”, and they can get really weird sometimes. Spotify used to return more of these in the official “genres” array returned for the Get Artist API, but now I think they keep those internally and the exposed ones are a lot more standardized.

In your own self-evaluation, how much did recommendations affect your music taste over time?

To be honest, not that much I don’t think? Nowadays I actively fight against it. I’ve been largely reducing my Spotify usage, and I pretty much buy all of my music on Bandcamp and listen to it just on my phone and make my own playlists. For more details about how I discover music, check out my post My evolving music discovery pipeline.

Are AI chatbots docs?

Wed, 09 Jul 2025 21:44:41 +0000

Robin Sloan asks: Is the doc bot docs, or not?

He relates a recent experience he had with an AI chatbot where the chatbot provided a response for his question that was incorrect despite being provided from the authoritative context of the official Shopify documentation.

Reacting, Sloan considers:

I suppose there are domains in which just taking a guess is okay; is the official documentation one of them?

I vote no, and I think a freestyling doc bot undermines the effort and care of the folks […] who work […] to produce documentation that is thorough and accurate.

As a writer of official documentation, I agree.

Most documentation sites are part of the official service agreement for a product, often serving as the warranty for how the product is supposed to work, and which uses are supported by the company. Meanwhile, most LLM technology is prone to hallucinations (not to mention prompt injection), even if constrained by the system prompt.

If you provide an AI-powered chatbot on your documentation site, you must consider how to make it a productive and accurate experience for your customers…

Imagine providing a more powerful search for your documentation, but instead of a search returning no results, it summarizes a best guess. Sometimes that can work, but sometimes that means you provide irrelevant information. The nature of generated text is that it seems far more authoritative than a clearly irrelevant link in a list.

It’s easy to add a bot to your docs site as an experiment, but if you do so, it’s important to be intentional about it, and follow up after deployment. If you want to add an AI-powered chatbot to your documentation site, before and after you add it, consider the following:

Add disclaimers

If the content output by your AI-powered chatbot might be inaccurate, consider indicating that your bot is in beta, or otherwise not officially supported.

If the service or license agreement that customers sign for your product includes your documentation as an official guarantee of product functionality, it might be a good idea to check with your legal team (if you have one) to ensure that the responses from the AI-powered chatbot that you deploy are not subject to the same guarantee.

In addition, you might want to add an information disclosure as well, letting people know how and whether the information they provide to the chatbot is collected and used.

Strategically supplement responses

Presumably, if you’re providing an AI-powered chatbot on your documentation site, it’s configured to supplement and source responses from the official documentation using retrieval-augmented generation (RAG).

When you identify the sources to use, you might want to consider whether those sources should include more than just the official product documentation. If you have a community forum site full of customer context (but less up-to-date or accurate information), or a technical blog full of use cases (and similarly potentially outdated), you might want to make those available to the chatbot.

Ideally you’d have some way of adjusting different weights (though it might be limited to temperature or other vague setting names). If you have multiple sources that you want to draw from, being able to indicate which ones are more authoritative than others, or at least prioritizing more recently updated content with the expectation that it’s going to be more accurate.

Test outputs

Set up automated and manual testing of the outputs for accuracy and consistency. The best way to do this is to build a “ground truth” dataset of questions and answers. After you have a canonical dataset of common questions and accurate answers, you can evaluate the output of the AI chatbot against that dataset.

Perform evaluations generally (is the output correct?) and over time (is the output still correct?) to ensure that the quality of responses doesn’t drift.

If you’re new to evaluating the output of LLMs, many chatbot providers like Kapa provide resources that outline these tactics, such as in Kapa’s doc Conversation review best practices. Intercom seems to provide a set of automated performance metrics for their FinAI chatbot which, if you trust them, you can use to review the quality of responses.

If you’re not using one of these providers (or even if you are), the field of data labeling, especially for NLP data, has a wealth of resources available. Microsoft provides a detailed list of metrics for evaluating LLM-generated content.

You can also automate testing of outputs with a tool like Evidently, incorporating testing into your CI/CD pipeline.

Monitor input and output

Beyond testing the quality and accuracy of the chatbot responses, you also want to monitor the questions being asked and the responses offered. The data from AI chatbot interactions can be used almost like a combination of search term data and community forum content — identify missing content, misaligned mental models, inconsistent terminology, customer use cases, and more that you can use to improve your content.

The monitoring process isn’t just about improving the accuracy of your content, but also its relevancy to customers.

Review customer inputs to understand use cases that customers have for your product, including the terms that they use to describe functionality.
Review the outputs from the chatbot to identify situations where the chatbot hallucinates information or mixes up concepts in its answer.

You can then take action in your regular documentation process to address anything that you find:

Enhance existing documentation with new end-to-end examples that draw from customer use cases.
Update content to include different terms that customers might use to describe particular functionality.
Create content to address missing functionality, such as by adding missing documentation about supported functionality, by posting an explanation about the unsupported functionality on the community forum, or something else.
Perform a terminology audit and update informed by unclear responses from the chatbot, identifying and resolving cases where you might be using the same term to refer to different concepts or functionality in your product.

Chatbots aren’t docs, but you can use the interactions with them to make your docs better.

Data maintenance matters

Thu, 29 May 2025 22:25:39 +0000

Data requires maintenance to stay accurate. For machine-generated data, monitoring the structure to ensure you’re collecting valid records is probably enough maintenance. For other types of data, however, it’s a bit more involved. Geospatial data, for example.

Molly Taft, writing for Wired in Trump Cuts Are Killing a Tiny Office That Keeps Measurements of the World Accurate shines a light on the National Geodetic Survey (NGS), which is responsible for maintaining spatial data for the United States of America:

NGS is currently responsible for maintaining and updating what’s known as the National Spatial Reference System, a consistent system of physical coordinates used across federal and local governments, the private sector, and academia. This includes not only latitude and longitude, but also measurements of depth and height as well as calculations around Earth’s gravitational field—crucial mathematics that inform much of the basic infrastructure around us, from constructing bridges to mapping out water and electric lines.

The Earth isn’t static — it’s constantly moving (remember, our land masses are basically just floating on magma!) and our data need to adapt to that movement. Modern engineering demands more precise data, and thus more precise measurements.

If you want to build a high speed train network across miles of desert, you need to know exactly where to lay that train track and build the bridges — not just in each acre but across the entire length of the rail line.

Measurements don’t just appear, they have to be made, and then accurately modeled to form an accurate spatial representation that is usable across the country (and continent). Who makes these precise measurements and builds the models? Geodesists. Unfortunately, the United States hasn’t spent much on their geodesy prowess:

In 2022, a group of leading geodesic experts authored a paper on what they dubbed the US’s “geodesy crisis,” detailing how other world powers have invested in training geodesists over the past three decades while the US has wound down funding and training. China has invested particularly heavily in creating more geodesists: the country graduates between 9,000 and 12,500 geodesy students per year, many of whom are then employed by the government.

Without these experts, what you’re left with instead is high demand without expertise:

Many industries now “want that high accuracy positioning” that comes with advanced geospatial technology, [Dave] Doyle [a former chief geodetic surveyor at NGS] says, “yet they don’t understand the basics of the science. Now you’ve got all these people punching buttons and getting numbers, and only a tiny percentage of them really understand what the numbers mean, and how one set of numbers relates to another.”

Whenever you encounter something built with data, remember that someone was there in the background, crafting and curating that data to ensure its accuracy, relevance, and utility. When we ignore, devalue, and defund these data workers, the quality of our data is diminished, and so too is the quality of anything that we build with that data.

Via The Map Room blog, in their post NOAA Cuts Threaten Spatial Reference System Update.

Where search keyword data comes from

Tue, 27 May 2025 23:27:51 +0000

As someone who does a lot of armchair-yet-informed speculation about how various things get defined in music metadata, this deep dive into how keywords and entity relationships have been formed by Google over the years was fascinating to me.

Deborah Carver, based on her years of expertise in the content industry, explores where SEO keyword data comes from in CT No.112: Demystify your algorithms: The origin of keywords:

If you receive a list of target keywords or a keyword research spreadsheet, do you know how they were sourced? Where exactly does your keyword data come from? Your answer is likely “from my browser extension,” “from our audience editorial team,” “from my keyword research tool” or “from our digital marketing agency.”

But of course, those keywords had to come from somewhere else, first! She details the different tools that Google has offered over the years, and defines the different datasets that contribute to the keywords identified by Google—including those that SEO tools add on.

As she points out, it’s advantageous to Google because these investments in keywords help drive the success of its primary revenue driver:

Giving away raw data and free training on its software has proved wildly profitable for Google. It’s a juggernaut business strategy in the open source tradition. The more people who learn to use keyword data, the more people understand search engine marketing, the more ads Google can sell.

Perhaps the most fascinating aspect for me, as a fellow content professional, is her clarification about the different ways that people use search keywords compared with how they use natural language:

Keyword research also helps stakeholders understand that people don’t use marketing speak in their search terms. People also don’t search with the same words they use to talk or chat or tweet. Imagine if, when suggesting dinner, your partner said, “pizza near me now.” No one talks that way.

Many people search that way—17,800 times each month across the globe, on average. Search language is its own behavior, one that’s ever-evolving.

I don’t think that “search language” as a behavior will change anytime soon, even with the rise of LLMs. Google and other search engines are adding LLMs to the results and processing of search queries in a purported effort to make searching “easier” and “more natural”, but so far, it seems to me that adding LLM output to search results is diluting the quality of results as people realize their trained habits for looking for information no longer work the same way.

And too, directly using an LLM to produce information requires specific language techniques to get better results. The same way search language has evolved over time, prompt engineering is emerging as its own type of language. Whether prompt engineering is a skill that, like search language, everyday people will need to pick up remains to be seen.

The technology might be changing, but at the end of the day, we’re all just learning new ways to speak to machines.

On the objectivity of data-driven decisions

Mon, 26 May 2025 21:30:43 +0000

Back in 2021, writing about the seemingly-objective Supreme Court Justices making politicized decisions, Benn Stancil interrogates the idea that data-driven decisions are equivalent to objective decisions in Tilt and tilted:

Making arguments from data, like interpreting the law through legal deliberation, isn’t inherently problematic. Quite the opposite, in fact—to the extent that it’s possible, data should be foundational. But, also like the law, it’s a foundation built on less level ground than we often admit.

Though we think of data as irrefutable ground truth, it is, in fact, also almost “entirely self-referential and made up.” Often, data—and its computational cousin, the metric—isn’t an abstract representation of an innate natural quality we’re attempting to quantify; it’s an accounting identity.

Unfortunately, the common perspective of using data to make decisions (or do anything) is that data is the truest form of objectivity:

To use data is to be level-headed. The surest sign of fairness is to support your claims with numbers; the surest sign of prejudice is to fail to do so. Data is both a sword and a shield: It is a weapon for prosecuting your point, and a defense for protecting yourself as reasonable and impartial.

But bias isn’t something that can be easily removed:

For those of us who work with data, the solution isn’t to make our data or analysis more objective. We can’t. Our raw material is too tilted, as are we.

It’s very common to see data as objective, but it’s important to remember that it isn’t any more objective than we are.

My writing setup

Sun, 25 May 2025 22:35:28 +0000

I do almost all of my writing in Google Docs and VS Code. Occasionally I’ll take longhand notes or jot things down in Apple Notes, but the real content that I write at work and for this blog is in one of those locations.

My VS Code setup

When I’m working in VS Code, I use several extensions to help make my work as easy as possible.

Code Spell Checker

Misspelling things in strings in code and when writing in an IDE without built-in spell check can be a challenge. This extension adds a straightforward and easy-to-use and configure spell checker to VS Code. It also advises on some grammar rules, non-intrusively.

Code Spell Checker from Street Side Software

Markdown All in One

This extension is best-in-class for writing Markdown. Get autocomplete for links, auto-created table of contents, and automated renumbering of lists.

There’s probably more in there, but those are the game changing features for me of this extension.

Markdown All in One from Yu Zhang

Markdownlint

Like any good linter, this extension helps enforce style standards when writing Markdown, ensuring that your syntax is consistent and correct throughout your documents. Also critical? It highlights when link fragments (like anchor links) are invalid.

I don’t like to see linting rules while I’m writing on a specific line, so I use the following settings to better manage that:

"markdownlint.run": "onSave",  
"markdownlint.focusMode": 5,

markdownlint from David Anson

TODO Highlight

If you add TODOs to what you write, you need to be able to find them again so that you can fix them.

TODO Highlight by Wayou Liu

Indent-Rainbow

After writing in ReStructured Text without a linter for over a year, and with any code that I write being in Python, being able to see exactly what indent level I’m at is crucial for valid markup and syntax.

indent-rainbow from oderwat

WordCounter

I forgot I had this extension installed because it’s unobtrusive. I use it to get a quick glance of how many words I’ve written for a blog post (and provide an early indicator that I’m writing an epic by accident).

WordCounter by Etienne Faisant

What I don’t use

I don’t use any git extensions, and I don’t use Vale.

I have GitLens installed but I’ve never actually used it because I prefer to use GitHub Desktop as my primary interface with changes that I make with git.

I don’t use Vale because I don’t use a style guide when writing my blog posts. When I’ve worked in docs-as-code environments, the style guides hadn’t been codified into Vale rules and programmatic style checking was not a priority.

Other VS Code settings

To make sure my files always get saved:

"files.autoSave": "onFocusChange"

To manage switching between git branches even easier:

"scm.workingSets.enabled": true

To make the text that I write easier to read:

"editor.wordWrap": "on"

To see every single space in my writing and eliminate accidental double spaces:

"editor.renderWhitespace": "all"

This was especially critical when writing in ReStructured Text, because an extra space could break a list or disrupt the formatting of an entire doc. Markdown is more forgiving, but I’m in the habit of enabling this setting ever since.

Google Docs settings

These are a few settings in Google Docs that I use all the time. Most of these can be set by going to Tools > Preferences:

Disable show spelling and grammar suggestions. I find it distracting to have squiggly lines appear when I’m typing a super rough draft or taking notes.
Disable automatically correcting spelling and capitalizing words. I find it especially distracting to have my words change as I’m typing them. I want typing in a Google Doc to feel almost as seamless as writing out longhand, and removing autocorrect from that process is vital.
Disable smart quotes. They’ll screw up any formatting that you copy and paste into a code sample, and I don’t need fancy typography.

I also use keyboard shortcuts constantly, whether it’s reformatting text to be a header or to match a standard style, or to paste without formatting.

I’ve been delighted with the Copy as Markdown and Paste as Markdown options. Despite the exact markup not being what I always choose to use, it makes formatting documents for tech review or codifying hasty drafts for publishing a lot faster.

Other shortcuts for writing

Generally, on my computer I tend to have a few keyboard shortcuts set up for the purposes of text autocomplete:

A text replacement rule to spell my first name correctly. Sometimes I type too fast and misspell it, and that would be embarrassing.
A text replacement rule to spell my company name correctly. Almost no matter what the company name is, it’s nice to have the failsafe.
A text replacement rule for my email address. It’s faster to type @@ than it is to type my full email address.

I’ll also set up other rules to catch common misspellings that I make — endpoing, documetnation — as well as finish long words or phrases — authz for authorization, authn for authentication, descr for description, and more.

I’m not big on personal optimizing, but I do find it helpful to make the process of typing as effortless of a translation from mind to document as writing longhand notes for myself.

My favorite music and more from 2024

Wed, 01 Jan 2025 11:05:08 +0000

I did most of my year-end discussion in my post How did my listening evolve? Spotify Wrapped 2024, but I wanted to spare a post for my favorites (subjectively, not quantitatively) of the year.

As a shortcut, I put most of the items on my lists that are available on Bandcamp in this Buy Music Club list.

Best albums of the year

SWIM - In Circles (95 listens)
- This scratched an itch in my brain that I can’t quite explain, but SWIM was one of my most-listened-to artists of the year (2nd, with 163 listens) and this album is a big reason why. Each SWIM track in my library averaged about 4 listens per track, which is pretty high. I thought I didn’t discover him until this year, but I actually first heard him on November 16, 2020, listening to his track Love (which is a total banger). At the time, he only had 4 songs out, although he released an EP a few days later.
- RIYL: The Blaze, Tourist
- Tracks: In Circles, Never, Condition
Caribou - Honey (71 listens)
- This album might be on the list mostly due to the show and boiler room set. If you have the chance to see Caribou play live, don’t miss out.
- Tracks: Volume, Come Find Me, Dear Life
Beyoncé - Cowboy Carter (130 listens)
- I’m not a country fan, mostly due to lack of exposure and interest, but I do listen to a good amount of folksy pop (? rock ?) like The Head and the Heart or Mumford and Sons, and I love a good pop song. Following on from her electronic album last year, this album kept the banging beats. With 27 tracks (at least 7 of which could be classed as interludes), there were several songs that didn’t hit for me. As a friend pointed out, the album felt more like a tour of country music than a tightly packaged album that was internally consistent. And that’s a good thing, mostly. I didn’t enjoy the Jolene cover, nor the song with Willie Nelson, but a lot of them really stuck with me. This album has skips, sure, but too many bangers to miss the top 5.
- Tracks: Sweet Honey Buckin’, II Hands II Heaven, Texas Hold Em, 16 Carriages
Tinashé - Quantum Baby (41 listens)
- It’s nice pop music, what else can I say. She’s much more than Nasty.
- Tracks: Nasty, Getting No Sleep, No Simulation, No Broke Boys
Tourist - Memory Morning (98 listens)
- Like several other albums on this list, at least 4 tracks on this album were released before the album, which is nice because they’re excellent, but annoying as well because that meant the album itself only had 5 or 6 new tracks.
- RIYL: Bonobo, O’Flynn, Catching Flies
- Tracks: A Little Bit Further, Siren, EST, Valentine
Jamie xx - In Waves (35 listens)
- Jamie is a reluctant 6, mostly because while this album didn’t initially hit for me, it got there in the end for me. I think it definitely suffered from anticipation and the overfamiliarity of the early-release singles. After some separation, I could treat it more like catalog and I can respect it for what it is, which is pretty good.
- Tracks: The Feeling I Get From You, Still Summer, All You Children

As an honorable mention, I have to put Waxahatchee - Tigers Blood on this list because I’m listening to it while I finish writing this. I only really heard about this album from Said the Gramophone’s Best of 2024 annual list, which always introduces me to some gems I wouldn’t otherwise find.

Best EPs of the year

A list, with some vibes descriptors.

DJ Heartstring - In Your Arms
- bouncy architected dance beats, an EP of three straight upbeat rave trace-y bangers.
Cameo Blush - Just Concentrate
- fuzzy driving soaring beats, as soon as 4 Me played I was a goner.
O’Flynn - Tensoon EP
- dancey thumping beats that make you want to spin. Count You In is an aggressive banger but don’t sleep on Medjool (I whisper “dates” to myself every time I read the track name).
Fennec - Nice Work Vol. 3
- groovy house and lo-fi beats, I kept coming back to the tracks on this EP, especially Better.
Baltra - Breathe Deep
- super solid airy floaty groovy beats, they can’t compete with last year’s track Luv-N-Me for perfection but a great release. Favorite track: Good Intentions.
Upper90 - DREAM CATCHER
- ravey trancey bangers, thumpy and dark, I don’t even know how I discovered this artist but it’s such a fun time. I like the first two tracks better than the last two.
Jeigo - Fig
- One of my friends described this EP as “we have Tourist at home”, which, true and probably explains why I enjoyed it so much. I kept coming back to the title track.
exxy - Heaven on Earth
- I waffled on where to place this one because I don’t think I really listened to the full EP until this month but the emotive nature of it (reminiscent of the xx in ways) made it resonate more with me. Favorite track: Trust (Don’t Let Me)

On the bubble:

t e s t p r e s s - losing control ep
Tourist and Gold Panda - Us Two
Tom VR - Acheless VIP
PLUS ONE - Ar Ais Aris Vol 2
isGwan - Heavy on the Drums vol 1
Giulia Tess - tempo00

Best tracks of the year

Some patterns you might notice in my favorite tracks: fuzzy bass, built up intensity that opens up / empties out into something anthemic. Also pop edits sorry not sorry (not ever sorry). After the first 4, the rankings are mostly arbitrary.

Once Twice - Nothing’s Perfect
- a perfectly transformed edit of Beyoncé’s Love On Top, but knowing that almost spoils the joy (sorry)
- adjectives: crunchy
Silva Bumpa - When I’m With You
- I want to dance along to this track every time I hear it
- adjectives: melodic joy, groovy footstepping
Club Angel - Cry
- A build of a track, it soars and makes me turn into a fist bumping foot stomping bro when it hits.
- adjectives: four-on-the-floor, persistent, whining but in a plaintive way
Tom VR - Caroline Polachek - Pretty In Possible
- Bass-driven Caroline is even better imo.
- adjectives: deep, haunting, stretched, layered yet empty.
TC4 - Joy dub
- I discovered this artist by shazamming a track during Sammy Virji’s set at 1015 Folsom and then digging into their expansive back catalog.
- adjectives: open drums, depth,
Bicep - CHROMA 001 HELIUM
- I accidentally bought this song twice. Bicep was my most-listened-to artist the last time they put out an album, so it’s maybe not surprising I loved their first entry back this much.
- adjectives: glitchy
TC4 - Hurt you
- Very similar structure to “joy dub” but a different vocal sample
- adjectives: haunting, open drums (is it the same sample as joy dub? lol)
Laurence Guy - Hey Baby
- I was like “this song is pretty good” until I heard it played out and then I was like whoa this song.
- adjectives: repetitive, cloying, driving
DJ Heartstring & SWIM - Alone Again
- I got really into ravey trance this year I guess and two of my favorite artists of the year on a track had to be good (and it was).
- adjectives: soaring, energetic
SHEE - Close My Eyes
- Okay so like if you listen to all of these songs in a row you start to hear the type of my music that I like.
- adjectives: bumpin, anthemic
Diffrent - A Little Closer
- Diffrent busting onto the UKG scene like the koolaid man with this sample-based banger. Interplanetary Criminal played it out a bunch before it came out and the b-side is also good.
- adjectives: garage, garridge
Joy Orbison - flight fm
- I started to think this track might be overrated but then I listened to it again and, nope, still great. Looking forward to coming back to it when it’s not the most overplayed electronic song of the year that isn’t by fred again..
- adjectives: buzzing, driving
Cameo Blush - Please Tell Me Why
- Satisfying combination of dark deep beats with soaring yet pleading vocals, a head bobber of a track.
- adjectives: soaring, bouncy, layered
bastiengoat - Let me see ya
- A bit of a departure from the rest of these tracks, this track feels more like a conversation in a living room.
- adjectives: intimate, bouncy, lo-fi
Floating Points - Birth4000
- I didn’t even realize this was based on the Donna Summer track until embarrassingly late. This track has been inextricably linked with flight fm for me for awhile because they both have similar builds and reveals imo. I didn’t love the album, but this track is still a standout.
- adjectives: fuzzy, satisfying, takes you on a journey
Laurence Guy - Most Perfect People (Are Mostly Not Perfect)
- a joyful house tune with echos of jangly guitar, I almost saw him play this year but didn’t make the effort to see him open (it was at the midway it was far okay).
- adjectives: twangy, jangly, soaring, headbobby
Tom VR - You’re Making My Teeth Grow Longer
- My teeth feel the same but my ears don’t. This kind of not ambient but drone-y type music has been a closet enjoyment of mine for a couple years (see also: Aloka, Surusinghe).
- adjectives: bouncy, pop rocks, deep, throbbing, sideways, persistent
Whethan & Elley Duhé - Money on the Dash (Marlon Hoffstadt Remix)
- Trancey rave bop, just try not to sing along.
- adjectives: daddy, kitschy, eurodance
Dresscode & Sudoo - Usually
- It’s a catchy track that does the whole layered then open then layered pattern that really hits for me. The sort of song that might make me dance in a circle.
- adjectives: driving, thumping, persistent
DJ Counselling - It’s Perfectly Clear Right Now
- This artist just kept showing up on my SoundCloud weekly playlists and kept being good so here we are.
- adjectives: fuzzy, cloying, drone

Best shows of the year

Caribou at the Fox: Best crowd, great set, excellent evening.
Sammy Virji at 1015 Folsom: The most energy I think I’ve ever seen at 1015 in all the years I’ve been going to shows there.
Bombay Bicycle Club at the Fillmore: The only lowlight was not getting a poster at the end of the night because they didn’t sell out early enough. So grateful they have a live album. So grateful they still play songs from the early days. Made me miss going to rock shows.

Best book of the year

Frostbite by Nicola Twilley, which is a fascinating and excellent social history, sorry not sorry if you encountered me during the weeks I was reading this.

Here’s a few quotes I loved:

controlled-atmosphere cold storage turned apples into widgets, and the only way for widgets to compete is on price.

Futures markets began with corn, wheat, and soybeans—commodities that could easily be stored for months in silos or sacks. With the advent of refrigeration, these kinds of contracts soon became common in the egg and butter markets too.
The commodity Hackett told me he would most like to trade is potato futures, a market that used to exist until J. R. Simplot, the billionaire credited with commercializing frozen French fries, defaulted on his contracts—“ just because he got on the wrong side of the market.” “The exchange allowed him to do it,” Hackett explained. “And it ended the futures market right there and then.”

In 1900, only 5 percent of married women worked outside the home; by 1940 that number had risen to 36 percent, and by 2000 it was 61 percent.

Other 2024 highlights

I didn’t blog a lot in 2024, but I wrote some big posts:

I wrote a post about the (broken) promise of the docs-as-code ecosystem, Docs as code is a broken promise and talked about it further in a Coffee and Content webinar of the same title.
I put together a career bucket list for technical writers that you can use to inspire your future in the industry.
I wrote 12K words about my listening patterns this year, and the evolution of Spotify Wrapped, in How did my listening evolve? Spotify Wrapped 2024.

My evolving music discovery pipeline

Wed, 01 Jan 2025 10:30:10 +0000

Over the years the dominant genres in my music taste have evolved, from mostly pop-punk in the early aughts, to primarily indie rock and bloghouse music in the late aughts, to house in the early tens, and now a much wider variety of electronic music, including UK garage and some trance.

As what and how I listen to music has changed over the last 20 years, the way I discover music has too.

Digital streaming platforms like Spotify are so pervasive now that it’s almost hard to remember the days when I was discovering music primarily by listening to the radio and reading music blogs, along with the occasional visit to the CD section of my local library.

5 years ago, algorithmic discovery via streaming services was a primary part of my music discovery process. In 2019, I wrote:

These days my music discovery is now largely driven by 3 services: Spotify, Shazam, and Soundcloud. Spotify pushes algorithmic recommendations to me, Shazam enables me to discover what track the DJ is currently playing when I’m out at a DJ set, and Soundcloud lets me listen to recorded DJ sets as well as having excellent autoplay recommendations. In all of them I have lists of tracks that I may never revisit after saving them. Some of them I’ll never be able to revisit, because they’ve been deleted or the service has lost the rights to the track.

Nowadays, my music discovery approach is somewhat more community-oriented, but still supplemented by many of the same tools I was using 5 years ago. One risk of those tools and the algorithmic discovery on streaming services is falling into an endless loop of discovering tracks, liking them, and moving on to discover more tracks.

I enjoy discovering music. For me, discovering music isn’t so much a process as a practice, something that I actively continue to do because it brings me great joy and also because I read at some point that most people form their music taste in their teens and never expand it, and I was determined to be different. But I also discover new music because I like music so much — which means I also need to take time to listen to what I’ve discovered, instead of getting caught up in digging up new new new new.

To try to balance out the allure of the new, I’ve tried to build a Bandcamp-to-offline-listening pipeline. Shortening the discovery to acquisition pipeline means that I can spend more time in the “burn in” period with a shorter list of songs (with the added bonus of being able to pay more money to more artists that I care about supporting).

Music discovery on Bandcamp

Discovering music through Bandcamp, in contrast to Spotify, feels more community oriented. Aside from the occasional algorithmically driven recommendation sent to my email, the site is entirely focused on a more self-driven exploratory approach.

When I listen to a track on Bandcamp, I can see the supporters of the track, any recommendations that they’ve made about their favorite tracks, and earlier releases from the same artist or record label. That’s another component of it too — independent record labels selling their catalog on Bandcamp, meaning that a discovery method that is basically impossible on Spotify (exploration by record label) And one that I haven’t had available since I was picking up sampler CDs at Warped Tour booths as a teenager is now a common and simple conduit for discovering new artists.

As a result, I end up discovering new music by digging into the back catalog of various artists, and more importantly, the record label publishing their music. Following a label to discover new music is a lot like following a promoter (instead of a venue) to discover new music. The label is going to release music that fits together, and a promoter is likely going to book more of the same music you like (or don’t know yet, but you will like) to play at a club. If you go straight to the source, the discoveries are a bit easier.

So there are more pathways to follow, ones that aren’t intercepted by algorithms at every turn, and I also build a closer connection with the artists and record labels I purchase music from. I get an email every time an artist or label that I follow releases new music.

Instead of hoping a new release shows up on my Release Radar playlist, or visiting a record store in person to find out about new music, I get a message in my inbox alerting me to the news. If an artist wants to announce a special show, some new merch, share a discount, or something else, they can also send a message directly to the list of supporters that Bandcamp has built:

It isn’t a personal message (it’s not written just for me—I already had a ticket!), but I nevertheless feel like I’m part of a network of fans, knowing and supporting an artist, instead of an enumerated follower or percentage-based fan on a data-driven dashboard somewhere.

This established community feel, where I can follow an artist (by letting them email me) rather than hoping that I find out about a new track because Spotify’s algorithm happened to add it to my Release Radar playlist that week, feels rooted in community and patronage, rather than a more passive feeling.

I don’t only listen to music that I buy, and trawling through Bandcamp isn’t the only way that I discover music. Some things haven’t changed — DJ sets are still a crucial part of my music discovery pipeline.

Music discovery from a live DJ set

When I go to a DJ set in person, I am the queen of shazamming while dancing To be less intrusive, I keep my phone on dark mode and sometimes also grayscale. . If I hear a song that I really like but don’t recognize, I’ll pull out my phone and Shazam it.

If Shazam doesn’t work, either due to lack of service or general inability to match the audio fingerprint in the database, I’ll take a 15 second or longer video (often of my feet) and keep dancing, saving the ID for later.

After the DJ set, I’ll go through the Shazams and build out a playlist, usually on Spotify, of the setlist. If I took any videos, I’ll also listen back and Shazam those in isolation to see if I can identify those tracks.

If I’m desperate to ID a song in a video, I’m in a Facebook group dedicated to identifying unknown tracks, or I can post in a discord of friends who are DJs to see if they have any leads, but it usually doesn’t come to that.

After I have a playlist, I’ll listen to it to validate whether or not the identified tracks are actually accurate. Shazam is a fallible tool, and oftentimes it identifies a particularly bassy track as something generic, or picks up an unreleased track as another song (because it has no audio fingerprint for the unreleased track yet). A giveaway is if you Shazam the same song multiple times and get different results every time.

In 2019 I wrote in my music newsletter about a notable Shazam mix-up, where it misidentified a Catching Flies song:

Shazam at one point picked up one of his tracks as “G#/Ab Pad by Drone Pads”, which despite being on Spotify, is pretty clearly stock sounds that could be purchased or that come with a set of pads, like Ableton Push. Artists can use these default sounds live to keep a set flowing continuously, or use software to enhance a performance in other ways.

Shazam falls down most easily when it tries to identify unreleased tracks and finds tracks with some of the same qualities yet are dramatically different. Some other examples:

Picking up Good Stuff Piano - Original Mix by Syap during a Disclosure set.
Sonata for two pianos KV448 in D Major, Andante performed by Alfred Brendel (written by Mozart) and C Pad by Drone Pads during a Jacques Greene set

After I test out the playlist, if I really like a song, or if I notice an artist in the list multiple times that I don’t know, I’ll go to their Spotify page and add the top 5 most popular songs, and sometimes also several of their most recent tracks, to a new playlist to dig deeper into the rest of the artist’s music.

At this point, I try to give the tracks some “burn in” time and listen to the playlist in the background a few times while doing something else. If any of the tracks catch my interest in that context, I’ll buy them on Bandcamp or wherever they’re available digitally, if anywhere.

Music discovery from a recorded set

I listen to a lot more recorded sets than live sets, for obvious reasons. I’ve been finding DJ sets to be a weird sort of pomodoro timer for me when I’m doing focus work, helping me sit down and stay focused by not having to curate a playlist to distract the loud parts of my brain (I’ve always been a person that focuses better with music on instead of silence) while also giving me checkpoints to take breaks, or to force myself to keep working for just a little bit longer.

This focus work approach is perhaps antithetical to the purpose of a DJ set, but it also means that I get to listen to a lot more DJ sets than I might otherwise, and therefore organically discover a lot more music than I might otherwise. A DJ set is curated, and the ratio of familiar songs to unfamiliar songs isn’t calibrated at all like a recommendation-based algorithmic radio station might be. Instead, it is designed to flow and build and react and drive emotion and dancing.

I listen to sets on either YouTube or SoundCloud, and on both sites I keep running playlists of sets to listen to and favorite sets. And here, once again, the community reigns supreme.

The best way to discover tracks in a recorded set is to read the comments. Sure, you’re not supposed to read the comments online, but the comments on YouTube videos of DJ sets are clutch. There’s often a 👑 comment MixMag profiled one such commenter, DJ Bigos, a few years ago. that has a full timestamped tracklist, compiled with the help of other commenters.

If I’m listening to a set on SoundCloud, occasionally the artist themselves will have posted a tracklist, but more often I’m relying on Shazam or sorting the comments by timestamp (which I recently discovered was possible!) to find comments around the time the song played and see if anyone has asked for the ID or posted the track ID already.

As a last resort, I can always put Shazam up to my headphone cans to get the ID. This is often a last resort for tracks that haven’t been ID’ed in the comments already, but it can pay off if the track is already released. Otherwise, I’m stuck making a note to follow the artist of the unreleased track This is what I did with a track in an I.Jordan Boiler Room set many months ago, where the producer himself replied to a comment asking for an ID with the name and release date of the track. , if there is one, and following up later.

After I have some track names to work with, I can dig deeper. I don’t usually make a playlist (why bother, if I can keep listening to the exact set itself) and instead go straight to Bandcamp or other sites to find the exact track that was played, or other recent tracks by the artist if the specific ID isn’t out yet.

I’ll still listen to the track to verify it sounds the same, that I still like it, and that it hits just as well outside the context of the set. If I don’t do this step very well, I can end up acquiring songs like The Birds by Question Mark, where every time it comes up on a playlist I wonder why I have the song… and then eventually it gets to the part that I liked a lot in a DJ set, and I relax into the track. But it’s almost 6 minutes long! Some tracks are built for mixing, not idle listening.

If I have enough time or like the artist enough based on the one track, I’ll click into their discography and dig into what else they’ve released. I did this with the artist TC4, who I heard at a Sammy Virji set in February (and the song isn’t available to stream in the USA and even the digital copies are sold out), but I dug deeper into their discography and fell in love with a few other tracks, and they ended up having a sale on their digital discography so I got the whole lot.

If there’s a lot of releases, I’ll pick a few newer releases to listen to, as well as some older ones, often based on the track name, cover art, or whatever strikes my fancy at the time.

I’ll listen through them and any that I especially enjoy I purchase and add to my phone to listen to while I commute.

Music discovery through online communities

I also discover music from friends and folks in online communities. Whether it’s the #music channel in my work slack, a text message from a friend who is really digging a new album, or the chaotically active channels of the Cloudcore Discord, recommendations from people are always a great way to discover music.

In some of these communities, I don’t know basically anyone personally, but we’ve been brought together by other shared interests — purchased music from the same record label, subscribe to the same email newsletter, or got to know the same DJ set promoters. The value here is in the diversity of the loose ties. The further afield the network, the less likely it is that I’ve heard the music they’re recommending.

If I’m lucky, some of them might even host a regular radio show that pipes those song recommendations directly into my ears (shoutout DJ sam3k and NO CHILL).

The flow of my music discovery pipeline

Because most of my music discovery ends with buying a track, EP, or LP on Bandcamp, I keep the pipeline going by following nearly every artist or record label that I purchase from, so I can add any new releases to my discovery pipeline.

While the volume of messages sent from Bandcamp can pile up and become overwhelming, it’s worth it to discover that an emerging favorite artist has released a new track, or an old favorite has put out an album without me noticing.

I do have to become my own personal algorithm, processing the information in each email with how I’m feeling at that moment (do I really want to listen to another deep house track right now? do I have the energy to discover if this new breaks and hard house artist is going to vibe for me or just stress me out?) to muddle through new discoveries.

My discovery habits are less driven by streaming services, but the platforms are still part of the pipeline.

Of the streaming services I use, the one that makes it easiest to discover music is SoundCloud, with their support of DJs posting sets and bootlegs, radio stations uploading mixes (thanks Rinse FM and The Lot Radio), and record labels often maintaining profiles on the site as well. Because you can repost tracks, you can easily explore from one artist’s profile to another to discover new-to-you tracks. There are algorithmically built playlists like Weekly Wave, but it’s also easy to dig deep into a rabbithole of new music.

I do still use streaming services to help others discover music by sharing links or making playlists.

A while back, I wished for better ways to share music with others:

Though so much of the web revolves around sharing now, it’s still a challenge to share music with others. Mixtapes and mix CDs are a thing of the past, and sending songs back and forth on Spotify is the closest you usually get to sharing your music taste with others.

Spotify deprecated in-app messaging in 2017, recognizing that letting folks use social media for sharing links was the future. Unfortunately, the availability and usage of universal sharing links is still pretty lacking, so you often end up with folks sharing Apple Music links with folks that only use Spotify sharing links with folks that only use Tidal sharing links with folks who primarily listen to tracks on YouTube sharing links with folks who prefer SoundCloud and so on. All the streaming services want you to share their music, but they’d prefer it to happen within their walled garden.

In 2019, I was still enamored of the collaborative potential of Spotify:

A primary reason for the success of Spotify for my listening habits is the social and collaborative nature of it. It’s easy to share tracks with others, make a playlist for a DJ set that I went to to share with others, contribute to a weekly collaborative playlist with a community of fellow music-lovers, or to follow playlists created by artists and DJs I love. My local [music] library can give me a lot, but it can’t give me that community interaction.

I don’t feel that way as much anymore. Maybe it was always a reflection of my in person communities, or not-yet-fractured social media groups primarily using Spotify, or just a shift in my own personal priorities as time has passed, but aside from one collaborative playlist with two longtime bffs through which I re-discovered how amazing caroline polachek is and which was partly responsible for my deep dive into pop music last year , I don’t get a sense of community or collaboration from Spotify anymore.

Instead, I find community (and discover new music!) in online communities, in person DJ sets, the comments of recorded sets, and appreciating the notes left by other Bandcamp supporters on various releases. Here’s to another five years of evolving music discovery.

How did my listening evolve? Spotify Wrapped 2024

Sun, 15 Dec 2024 09:55:21 +0000

It’s time for my annual deep dive into Spotify Wrapped! I continue to primarily listen to music I own and diversify my streaming habits beyond Spotify to include SoundCloud and YouTube as well. As a result, my Spotify Wrapped is less and less representative of my actual listening history.

Let’s dig into my top artists and tracks of 2024 according to each service, and dive into the Spotify Music Evolution categories.

You can use this table of contents to skip around:

Comparing Spotify with SoundCloud and Last.fm

I’ve been using Last.fm to track what I listen to since 2007. In this post I’ll compare the insights that I got this year from Spotify Wrapped and SoundCloud Playback with my Last.fm data. These data sources are not equivalent.

My Last.fm data is accurate to within minutes, so might even change as I’m writing this post, and tracks most of what I listen to, no matter what I used to listen to it (with some limits).

SoundCloud has a help article, Your Playback playlists, that doesn’t list a time range for 2024 as I’m writing this, but lists the time period covered by SoundCloud Playback in 2023 as from January 1 – November 30 for 2023 and earlier, so I’ll comfortably assume that the 2024 version covers the same time period. The Playback, of course, covers only what I listen to on SoundCloud.

For the first time ever, Spotify Wrapped, according to the Spotify Wrapped article on their support site, uses data from November. They’re very vague about it, however:

The 2024 Wrapped personalized experience covers the music and podcasts you streamed from January 1st, 2024, to a few weeks prior to launch on December 4th, 2024.

I’d guess that the cutoff is around November 15th, informed by my own data and an uncited Wikipedia edit that I spotted while researching this. To make this happen, some engineers at Spotify probably improved their data processing infrastructure, or they reduced the overall insights delivered in order to capture a longer time range. Either way, Spotify Wrapped only includes data from Spotify.

With those caveats, let’s do some rough analysis.

Dig deeper into your own Spotify Wrapped

If you’re reading this and wish you had something like Last.fm to dig into your own music listening habits (and you also listen mostly to music on Spotify) then you’re in luck.

Thanks to privacy laws and regulations like GDPR and CCPA General Data Protection Regulation and the California Consumer Privacy Act , you can request a download of your streaming history from Spotify. According to the site, the privacy request takes at least 5 days (for the past 1 year of streaming history) or up to 30 days (for all time listening history) to return.

Ordinarily, that data would be kind of annoying to process (unless you, like me, work for a data analysis software company, or less like me, are extremely good with spreadsheets), but glenn mcdonald Former Spotify data alchemist (hired on as part of the Echo Nest acquisition), he was one of the 1500+ people laid off by Spotify in layoffs announced last December (more on those later in this post). built a tool called Curio to help you make your own Spotify Wrapped. Announcing it in a post titled Know Yourself (For Free), he says:

It’s that time of year when companies begin pretending that a) the year is already over, and b) you should be grateful to them for giving you a tiny yearly glimpse of your own data.

But your data is yours. You shouldn’t have to elaborately ask for it, but that tends to be the state at the moment. You can get your streaming history from Spotify by going to Account > Security and privacy > Account privacy (of course).

But if you do, because you can, and you get a Spotify API key, which you can, then you can play with Curio, my experimental app for organizing music curiosity. Curio does a potentially puzzling assortment of things that I like to do, but it also has a query language, so that neither of us is limited by what I already think I want.

I’ve requested my data and I’m looking forward to exploring the lenses that Curio might enable me to see, as well as to possibly disaggregate my listening data by service.

By the way, other services like Shazam or SoundCloud Also the grocery store chain that I shop at, in case you also want to perform year-over-year grocery cost savings analyses (stay tuned for a future blog post I hope). provide this ability too. Look for phrases like “privacy request” or “request to know” for details.

My top artists of 2024

I didn’t have strong expectations about my top 5 artists on Spotify this year because I’ve been trying to use it less and less over the past few years. I did have some guesses, though.

When you evaluate music data based on frequency, what you also reveal is different patterns of how you “use” music in your day-to-day life. My partner listens to a lot of music while exercising. One friend of mine often texts me for recommendations for tracks to add to his bike playlist. Another friend plays music for her dogs.

I focus better with music and end up in a good number of “listening ruts” (see last year’s post on Bombay Bicycle Club) as a result, my listening patterns reflect various writing projects that I’m doing at work or at home (like this one!).

What were my top artists, according to each service?

Spotify	SoundCloud	Last.fm
Beyoncé	SWIM	TC4
U2	Jasper Tygner	SWIM
Charli XCX	Overmono	Beyoncé
Caribou	Fred again..	DJ Heartstring
Tourist	17 Steps	U2

There’s definitely some surprises here, and some embarrassing admissions.

What is streaming for?

Beyond the different functions that music serves in my life, I also use different music streaming platforms for different purposes. My top artists list reflects this.

If I’m listening to music on a streaming service it’s because I want to listen to something I don’t own, or that I’m not using one of my personal devices.

Spotify is where I go to evaluate a new album by an artist, especially if it’s being talked about a lot (Beyoncé and Charli XCX) and where I go to listen to music that I don’t own but is more classic (U2). I also do still listen to music that I own, but mostly as focus listening (Caribou and Tourist).

On SoundCloud, I primarily listen to DJ sets, scout out IDs and remixes, and discover new artists and tracks. One artist from my top 5, 17 Steps, is actually a record label. I’m guessing that because of the way that profiles and metadata work on SoundCloud There are many more freeform fields—artists and labels and radio stations all have profiles, and the track name titles are derived from the uploaded name of the track, which doesn’t always match the actual official track metadata. This also makes it easier to upload and manage DJ sets and bootleg cuts and remixes, making SoundCloud a popular choice for electronic producers (and more) , it’s showing up as one of my top artists.

As far as I can tell, I’ve only liked one track from them and that track is one I’ve only listened to once, on May 11 of this year.

I might have listened to more tracks, but there’s no easy way for me to trace that lineage from my Last.fm data.

Either way, including that record label as a top artist is an odd choice, especially given that O’Flynn, Club Angel, and others show up in my top tracks in the Playback 2024 playlist (and I definitely spent at least one entire day only listening to most of the track son Club Angel’s profile on SoundCloud).

An unexpected surprise

I knew as soon as I saw my top artists in this year’s Wrapped that it had extended into November, because U2 was there in second place. I didn’t listen to U2 at all this year, and then after both a conversation where I’d convinced a friend (correctly) that The Joshua Tree was an excellent album and a few impressive appearances by U2 in the documentary Eno, I simply needed to listen to them:

And then something clicked in my brain and U2 was the only thing I wanted to listen to. No really:

I don’t want to shatter any illusions that you might have had about my music taste, but soaring anthemic music exists across genres, and The Joshua Tree is excellent.

What (and who) is missing?

There are some patterns that I expected to see in the data that are missing. Another purpose that I used Spotify for this year was digging deeper into the music mentioned in two music history books that I read:

The Number Ones by Tom Breihan, which explores the history of evolving taste through the lens of chart topping singles from 1958 to the present.
Let’s Do It: The Birth of Pop Music: A History by Bob Stanley, which covers the birth of pop music up until about the early 1960s.

As I was reading those books, I of course wanted to seek out the songs mentioned, and I was lucky enough to find a set of public playlists with nearly every track mentioned in “Let’s Do It” (there’s also one for the Number Ones, but it only covers the highlighted tracks, not every one referenced). In this way, Spotify served briefly as a music history experience for me — but none of that listening history appeared in my Spotify Wrapped or in my top 100 tracks, or in my “Music Evolution” playlist.

Another artist completely missing from my Spotify top artists is TC4. The artists in my SoundCloud and Last.fm top 5 artists of the year show up at least in my Top Songs of 2024 playlist, but TC4 is totally absent.

That’s because TC4 is pretty absent from streaming services as well. Their profile on Spotify is pretty empty, with only 26 tracks across 20 albums (singles) in their discography.

Meanwhile, on Bandcamp, it’s a different story. They have 93 releases, and while most of them are still 1 or 2-track singles, that’s more than triple the number of tracks available on Spotify. They also sell their discography at a discount, so I bought it in May of this year.

Even if I hadn’t fallen in love with a few of their tracks (Joy dub, Hurt you, Bandido), the sheer number of tracks in my library means that listening idly on shuffle often brings up one of their tracks.

What makes a top artist?

Another aspect of these top 5 rankings is assessing what “top” means in the context of these rankings. You can rank artists by volume of listens, but you can also weight volume with other factors, like popularity or consistency.

I am not a machine learning practitioner and I have never taken a stats class, but I do have lots of data, so let’s do an experiment!

Are my top artists according to each service ranked only by volume of listens?

Spotify	SoundCloud
Beyoncé	SWIM
U2	Jasper Tygner
Charli XCX	Overmono
Caribou	Fred again..
Tourist	17 Steps

The Spotify top songs playlist is ranked from most frequent to least frequent listens, so I considered that maybe the top artists would be reflected in it as well. The first five unique artists to appear in my top songs playlists from each service are the following:

Spotify	SoundCloud
Jamie xx	SWIM
Caribou	O’Flynn
U2	DJ Heartstring
Beyoncé	Bubble Love
Effy	Club Angel

I’ve bolded the top 5 artists from Spotify and italicized the top artists from SoundCloud that show up in this list.

There’s some overlap, but the order is different, which makes sense. Tourist and Charli XCX have been supplanted by Jamie xx and Effy for Spotify, and it’s a totally different list for SoundCloud aside from SWIM.

So definitely my top 5 artists are not just the first five unique artists who made my top songs of the year. What about the artists that occur most frequently in the top songs of the year playlist?

Spotify	Playlist frequency	SoundCloud	Playlist frequency
Beyoncé	12/100	SWIM	4/50
Tourist	10/100	Overmono	4/50
U2	6/100	Jasper Tygner	3/50
Bicep	5/100	Cameo Blush	2/50
Charli XCX	5/100	Club Angel	2/50
DJ Heartstring	5/100	DJ Heartstring	2/50
Jamie xx	5/100	HAAi	2/50
SWIM	4/100	Leon Vynehall	2/50
Caribou	4/100	O’Flynn	2/50
Bombay Bicycle Club	2/100	Ross from Friends	2/50
Chappell Roan	2/100	Totally Enormous Extinct Dinosaurs	2/50
Jeremy Loops	2/100	-	-
Malugi	2/100	-	-
Narciss	2/100	-	-
Tinashe	2/100	-	-

The Spotify list is pretty similar to my top 5 artists, but is again in a different order and lists Bicep instead of Caribou.

If I try to backsolve for the qualities of a “top artist” according to Spotify (I won’t even try for SoundCloud, because of their mysterious metadata), I’ll start from the same presumed base: listen frequency, or volume.

These are my top 20 artists of the year by listens, according to Last.fm:

artist	listens
TC4	212
SWIM	163
Beyoncé	146
DJ Heartstring	141
U2	139
Tourist	137
Caribou	107
Bicep	106
Chappell Roan	91
Jamie xx	87
Charli XCX	82
Fleetwood Mac	81
Mall Grab	80
Jasper Tygner	74
Bombay Bicycle Club	73
NOTION	67
Barry Can’t Swim	63
salute	62
Tom VR	55
Fred again..	54

To compare alternate versions of “top”, let’s also look at the total number of days in 2024 that I listened to these same artists:

artist	days
SWIM	66
TC4	62
DJ Heartstring	57
NOTION	45
Barry Can’t Swim	41
Tourist	40
Tom VR	39
Jasper Tygner	37
Bicep	36
Caribou	35
Charli XCX	35
Jamie xx	34
Mall Grab	31
salute	29
Fred again..	27
Chappell Roan	24
Bombay Bicycle Club	23
Beyoncé	17
U2	12
Fleetwood Mac	11

And finally, let’s collect the Spotify popularity rankings for each artist:

artist	popularity
Chappell Roan	87
Beyoncé	87
Fleetwood Mac	83
Fred again..	79
U2	79
Jamie xx	73
Charli XCX	67
Barry Can’t Swim	64
NOTION	63
Bicep	62
Caribou	59
Bombay Bicycle Club	57
DJ Heartstring	56
Mall Grab	56
Tourist	54
salute	54
Jasper Tygner	47
SWIM	46
Tom VR	35
TC4	18

My data is, as I mentioned, a different dataset from Spotify’s, but it is interesting to see how much the artists move around when ranked differently.

Are top artists biased toward popular artists?

Several folks I’ve talked to about Spotify Wrapped have speculated that it’s biased toward the popular artists. Looking at my top artists with these different rankings, it seems to me that popularity could be contributing to the ranking, and therefore the top 5 artists selection.

Let’s test this out with a basic linear regression I didn’t really know what this looked like in practice but thankfully I asked for help sorting through my ideas so now I get to say fancy things like this. …

If you don’t want to read the detailed analysis, feel free to skip to the findings!

Equal weights

First, let’s test out equal weights for listen frequency, listen consistency, and artist popularity:

(listens * 1) 
+ 
(days_listened * 1) 
+ 
(artist_popularity * 1)

This yields the following ranking:

artist	equal_weights
TC4	292
SWIM	275
DJ Heartstring	254
Beyoncé	250
Tourist	231
U2	230
Bicep	204
Chappell Roan	202
Caribou	201
Jamie xx	194
Charli XCX	184
Fleetwood Mac	175
NOTION	175
Barry Can’t Swim	168
Mall Grab	167
Fred again..	160
Jasper Tygner	158
Bombay Bicycle Club	153
salute	145
Tom VR	129

Using equal weights for each metric gets pretty close to the actual Spotify top 5, but I’m curious how it changes with uneven weights.

Popularity as more important

Next, let’s test out adding more weight to popularity, using 20% as a way to provoke an obvious but not unreasonably high change:

(listens * 1) 
+ 
(days_listened * 1) 
+ 
(artist_popularity * 1.2)

That calculation yields the following ranking:

artist	upweight_popularity
TC4	295.6
SWIM	284.2
Beyoncé	267.4
DJ Heartstring	265.2
U2	245.8
Tourist	241.8
Chappell Roan	219.4
Bicep	216.4
Caribou	212.8
Jamie xx	208.6
Charli XCX	197.4
Fleetwood Mac	191.6
NOTION	187.6
Barry Can’t Swim	180.8
Mall Grab	178.2
Fred again..	175.8
Jasper Tygner	167.4
Bombay Bicycle Club	164.4
salute	155.8
Tom VR	136

That chart looks basically identical to the original chart, with Beyoncé and U2 jumping up but many artists staying in roughly the same region.

Consistency as unimportant

Another consideration is whether consistency is actually a factor at all in determining the top artists. Let’s try downweighting it in the calculation, with other weights being equal:

(listens * 1) 
+ 
(days_listened * 0.8) 
+ 
(artist_popularity * 1)

These results look even more similar to the original top 5 list:

artist	downrank_consistency
TC4	279.6
SWIM	261.8
Beyoncé	246.6
DJ Heartstring	242.6
U2	227.6
Tourist	223
Chappell Roan	197.2
Bicep	196.8
Caribou	194
Jamie xx	187.2
Charli XCX	177
Fleetwood Mac	172.8
NOTION	166
Mall Grab	160.8
Barry Can’t Swim	159.8
Fred again..	154.6
Jasper Tygner	150.6
Bombay Bicycle Club	148.4
salute	139.2
Tom VR	121.2

Actually, let’s try just excluding consistency entirely:

(listens * 1) 
+ 
(artist_popularity * 1)

That calculation produces the following ranking:

artist	exclude_consistency
Beyoncé	233
TC4	230
U2	218
SWIM	209
DJ Heartstring	197
Tourist	191
Chappell Roan	178
Bicep	168
Caribou	166
Fleetwood Mac	164
Jamie xx	160
Charli XCX	149
Mall Grab	136
Fred again..	133
NOTION	130
Bombay Bicycle Club	130
Barry Can’t Swim	127
Jasper Tygner	121
salute	116
Tom VR	90

That looks much closer to my actual Spotify top 5. Let’s see if upweighting popularity is even closer. This time, I’ll upweight popularity by 30%, because it’s a proxy for newness as well:

(listens * 1) 
+ 
(artist_popularity * 1.3)

That calculation produces these final rankings:

artist	upweight_popularity
Beyoncé	259.1
U2	241.7
TC4	235.4
SWIM	222.8
DJ Heartstring	213.8
Tourist	207.2
Chappell Roan	204.1
Fleetwood Mac	188.9
Bicep	186.6
Caribou	183.7
Jamie xx	181.9
Charli XCX	169.1
Fred again..	156.7
Mall Grab	152.8
NOTION	148.9
Bombay Bicycle Club	147.1
Barry Can’t Swim	146.2
Jasper Tygner	135.1
salute	132.2
Tom VR	100.5

I think as long as I’m using the Last.fm data, I won’t be able to shake Caribou and Tourist out of their spots above Charli XCX. I bought both their albums, and Caribou had an excellent Boiler Room set to boot.

Correct for listening activity

What I can do is to adjust the rankings to account for what I know about my listening activity. The ranking with upweighted popularity and no accounting for consistency of listens seems pretty close to the Spotify top 5, so let’s start there, then assign corrections for artists that don’t fit:

artist	upweight_popularity	correction
Beyoncé	259.1	keep
U2	241.7	keep
TC4	235.4	exclude
SWIM	222.8	exclude
DJ Heartstring	213.8	exclude
Tourist	207.2	downrank
Chappell Roan	204.1	downrank
Fleetwood Mac	188.9	exclude
Bicep	186.6	downrank
Caribou	183.7	downrank
Jamie xx	181.9	downrank
Charli XCX	169.1	keep
Fred again..	156.7	downrank
Mall Grab	152.8	exclude
NOTION	148.9	exclude
Bombay Bicycle Club	147.1	exclude
Barry Can’t Swim	146.2	downrank
Jasper Tygner	135.1	downrank
salute	132.2	downrank
Tom VR	100.5	exclude

Any artist labeled exclude is one that I either didn’t listen to on any streaming services, or didn’t listen to on Spotify specifically, choosing to listen on SoundCloud or YouTube instead. My listening to Fleetwood Mac specifically took place outside the time window for Spotify Wrapped, so I’ll also exclude that activity.

Any downranked artists are those whose music I purchased, but still listened to on Spotify at least some of the time by my recollection.

Let’s remake the table, assessing a 30% tax to downrank the relevant artists accordingly:

artist	revised_rank
Beyoncé	259.1
U2	241.7
Chappell Roan	176.8
Charli XCX	169.1
Tourist	166.1
Jamie xx	155.8
Bicep	154.8
Caribou	151.6
Fred again..	140.5
Barry Can’t Swim	127.3
salute	113.6
Jasper Tygner	112.9

The listen volumes of Bicep and Jamie xx and the popularity and listen volume of Chappell Roan couldn’t be totally caught, but with this added adjustment we can pretty closely approximate the Spotify top 5 artists. Here’s the full methodology that I used to produce this result:

Start with a list of the top 20 most listened to artists according to Last.fm
Ignore any details about listen consistency
Weight artist popularity by an added 30%
Exclude any listening activity that occurred fully off Spotify
Downrank any listening activity that mostly occurred off Spotify by discounting it by 30%%

Or, as a formula:

(listen_frequency * if(rank = "keep", 1, rank="downrank", 0.7)) 
+ 
(artist_popularity * 1.3)

With these data correction and weighting tactics, I can effectively re-rank my most-listened-to artists and end up with nearly the same top 5 artists as Spotify.

I didn’t need to do a bunch of fake math to attempt to replicate my top 5 artists, but I wanted to, so here we are.

The anecdata agree

Some anecdata from friends support a bias toward popularity, but also some artists or tracks missing that they’d expect to see represented. Algorithmic weighting by popularity might account for some of that, but it’s also possible that Spotify does some filtering.

At first, I thought that maybe the filtering would be to weight by newness to favor artists that released music in 2024, but the popularity metric alone might be enough to weight by newness because newness directly contributes to the popularity score.

The documentation for the get_track API endpoint clarifies:

The popularity is calculated by algorithm and is based, in the most part, on the total number of plays the track has had and how recent those plays are.
Generally speaking, songs that are being played a lot now will have a higher popularity than songs that were played a lot in the past. Duplicate tracks (e.g. the same track from a single and an album) are rated independently. Artist and album popularity is derived mathematically from track popularity. Note: the popularity value may lag actual popularity by a few days: the value is not updated in real time.

Maybe if I’d listened intensely to a less popular U2 album, or to a more obscure album by another artist, my weeklong U2 deep dive wouldn’t have appeared at all. Or perhaps no matter the artist popularity, the listening intensity was too all-consuming to exclude.

For some people that shared their experiences, some clear consistent listening habits were missing:

Listening to Elliott Smith (popularity = 68) a lot after the election (which is when my U2 listening happened).
Listening to classic rock and folk music like Crosby, Stills, Nash & Young (popularity = 61) or Joni Mitchell (popularity = 64) regularly throughout the year.

And my personal favorite:

Listening to Monster Mash (popularity = 70) once every day.

For these people, these tracks and artists didn’t appear on their Spotify Wrapped at all (Monster Mash was supplanted by Chappell Roan’s track Good Luck, Babe, which has a popularity of 93).

It could also be that in an attempt to highlight new and popular tracks and make Spotify Wrapped feel like a true wrap up of a particular year, consistent listening to catalog music Released at least 18 months ago. might be downranked in favor of artists that were new to the user that year.

In 2022, Spotify did account for how much of a user’s listening activity was to catalog music or to new music as part of their Listening Personality assignments, so it’s not a new thing for Spotify to account for, algorithmically.

My top tracks of 2024

Now that we’ve dug into my top artists of the year, what were my top 5 tracks of 2024 according to each service?

Spotify	SoundCloud	Last.fm
Jamie xx, Honey Dijon - Baddy On The Floor	SWIM - ID - freddie’s island	Bicep - CHROMA 001 HELIUM
Caribou - Broke My Heart	Mixmag - Impact: Interplanetary Criminal	U2 - Running to Stand Still
Caribou - Honey	SWIM - ID - don’t	U2 - I Still Haven’t Found What I’m Looking For
Jamie xx - Treat Each Other Right	SWIM - ID - moveit	Jamie xx - Treat Each Other Right
U2 - Running To Stand Still	SWIM - ID - slowdown	Jamie xx - Baddy On The Floor

SoundCloud doesn’t provide a list of top tracks in its statistics, so these are the first 5 tracks in the Playback playlist.

The good news is that I enjoyed all of these songs enough this year to purchase all of them, except for the Mixmag DJ set with Interplanetary Criminal because it is not a song.

It took me a few days to decide that I should probably just buy U2’s Joshua Tree rather than listen to the same 4 songs on repeat the day after the election. I preordered both Caribou and Jamie xx’s new albums, and I was disappointed that both of them ended up releasing 4-5 tracks off the album as singles before the album came out, but that did make it easier for those singles to show up higher in my listening habits.

Given that Jamie xx was releasing an album for the first time in a while, a lot of my listens to Jamie xx were evaluative — do I like this track? do I like this new direction? — rather than listening to the songs because I already liked them. That led to a different kind of repeat listening than the U2 tracks in the top 5 did, which showed up because they were hitting the right brain groove at the right time.

SWIM released his IDs album after I had spent a few weeks listening to a few of his sets on YouTube where he was playing mostly those IDs, so when they came out I definitely consumed them, although they were released on Bandcamp first (where I bought them) before SoundCloud. I still listened to them on SoundCloud while at work, so I could see this prominence making sense.

My total listening activity on SoundCloud was only about half as much as on Spotify, so the playback is also drawing from a smaller pool of music to remind me about, and a lot of that pool is made up by DJ sets too.

While 2 of my top songs of 2024 according to Spotify were by Jamie xx, he didn’t appear in my top 5 artists of the year. Because his popularity score is higher than both Tourist and Caribou, I don’t think it’s a matter of popularity bias. A more likely explanation is that I preordered his album, so after his album came out I converted most of my listening activity to my personal devices rather than continuing to stream on Spotify.

My top song of 2024

According to Spotify, my top song was Baddy on the Floor by Jamie xx and Honey Dijon, which I listened to 16 times.

That listening activity put me in the top 0.05% of listeners worldwide. According to Spotify, I first streamed it on April 22, 2024, which is a week after its release date.

Let’s check in with Last.fm to see how that matches up:

I listened to the track 23 times in total.
I first listened to the track on April 21, 2024 at 17:03:45, so while I first heard it on Spotify on April 22, I’d already heard the song by then, likely on Bandcamp.
Spotify counts a track as “streamed” if you listen to at least 30 seconds of the track, so that could be part of why their count seems somewhat high here.

Perhaps I really did just stream it 16 times on Spotify while trying to decide if I liked it or not. I also recall a few times when the autoplay suggestion after a playlist finished was this track, which is somewhat of a self-reinforcing way to get to top song of the year — the algorithm made it so.

According to Last.fm, my actual top song in terms of total listens is CHROMA 001 HELIUM by Bicep, with 25 total listens.

This surprised me, but likely means that I’m extremely ready for Bicep to put out a new album. Just a few years ago, Bicep dominated my top songs of 2020.

How the playlist gets made

Given the different weights and contributors that might affect the top artists ranking, I was curious if similar factors might affect the “Your Top Songs 2024” playlist.

The Spotify Wrapped support page is specific yet vague:

Is the ‘Your Top Songs 2024’ playlist in order of what I listened to most?
Yes, it’s ordered by the songs you played most frequently to least frequently.

It isn’t confirmed that the top songs playlist is the exact songs that you played, from most frequent to least frequent, but the tracks that are included are in frequency order. It’s probably easiest to pull the data that way, but in an effort to make the playlist more varied and representative of someone’s year, there might be some amount of curation or filtering happening.

Given the prominence of SWIM and Club Angel on my SoundCloud Playback, I think SoundCloud is sticking to the basics.

If I compare the first 40 tracks or so from the Spotify Your Top Songs 2024 playlist to my most-listened to tracks on Last.fm, some evidence of my listening habits are revealed:

Tracks from albums that I evaluated on Spotify show up on the playlist, such as those from Beyoncé, Fred again.., Charli XCX, Caribou, and Tourist.
Tracks that I discovered while compiling DJ set playlists based on Shazams also showed up on the list.
Tracks that I discovered from other playlists, like the South African Jams playlist that Rian van der Merwe put together and shared on his blog in January, also appeared.

Meanwhile, the broader picture of my listening captured by Last.fm includes:

A post-November 15 deep dive into Fleetwood Mac’s album Rumours
Being dedicated to SWIM’s album and DJ Heartstring’s releases.
Enjoying the artists whose albums I bought this year, like Beyoncé, Caribou, Tourist, and Jamie xx.

Many of the same songs are in the top 20 tracks for each service, but at different rankings:

Total songs heard in 2024

Spotify says I listened to a total of 2,616 songs this year. According to my own records, I listened to at least 8,949 and counting. Even shaving off a few for potential duplicates or miscounts, that’s an order of magnitude larger. My attempts to listen to more music offline, and off Spotify, continue to succeed.

Total minutes listened in 2024

For each service, here’s how many minutes of music I ostensibly listened to:

Spotify	SoundCloud	Estimated total
14,248	6,055	59,948

For both Spotify and SoundCloud, it’s unclear to me if the minutes listened are the actual number of minutes that I listened to music, or a more basic calculation of:

(track_name * track_duration) * listen_count

I’d wager it’s the formula, if only because it’s simpler to calculate on the fly, but it really comes down to what level of granularity of user activity is stored and made accessible for compiling these stats. Spotify also counts offline listening, but I’m not sure if SoundCloud does (which, if not, many hours of DJ sets are missing).

As part of user session data, Spotify keeps track of whether you listened to more than 30 seconds of a track Researchers at Spotify published a dataset, The Music Streaming Sessions Dataset with an accompaning paper that mentions three different skip tracking behaviors: skip 1, indicating whether the track was only played very briefly, skip 2, indicating whether the track was only played briefly, and skip 3, indicating whether most of the track was played, as well as not skipped. . SoundCloud, on the other hand, counts a track as played “once the play button is clicked” according to Insights basics – SoundCloud Help Center.

My own estimated minutes calculation is based on an estimate made with the assumption that if I listened to the track, I listened to the entire track. This is partially based on the fact that a Last.fm listening event (a scrobble) is recorded if a track is longer than 30 seconds and “the track has been played for at least half its duration, or for 4 minutes (whichever occurs earlier.)”

To calculate an estimate of minutes listened, I use the following methodology. For each track that I listen to:

If a matching track exists in my music library, use the actual track duration.
If a match does not exist (which could be due to mismatched metadata), use a constant value equivalent to the mean average track duration of all matched tracks listened to in 2024.

For 2024, that mean average track duration is about 4.45 minutes, or 4 minutes, 27 seconds.

Notably, any DJ set or concert recording that I listened to on YouTube or SoundCloud is totally misrepresented in this data. For example, I listened to Chappell Roan’s Bonnaroo set at least 14 times.

With this estimate, those 14 listens would contribute about 62.3 minutes to my total for the year.

In reality, that Bonnaroo set recording is 58 minutes and 22 seconds long. Listening to that 14 times is 817.13 minutes — a difference of more than 13 times.

Returning to the Spotify Wrapped data, my biggest listening day was October 11, with 404 total minutes listened.

If I look at my overall listening behavior, and factor in the estimated time spent listening each day, I get the following:

day	listens	length (minutes)
Jun 10, 2024	113	503
Apr 03, 2024	110	490
Jun 25, 2024	108	481
Oct 30, 2024	105	467
Oct 04, 2024	103	458
May 01, 2024	93	414
Jun 05, 2024	93	414
Apr 23, 2024	89	396
Apr 08, 2024	87	387
Jul 10, 2024	86	383
Sep 06, 2024	86	383
Apr 05, 2024	85	378
Oct 09, 2024	84	374
Jul 23, 2024	83	369
Oct 10, 2024	83	369
Dec 04, 2024	82	365
Jan 18, 2024	81	360
Jun 08, 2024	80	356
Oct 11, 2024	78	347

October 9, 10, and 11 were popular days for lots of music listening, it turns out.

347 estimated minutes compared to 404 ostensible Spotify minutes isn’t bad, but let’s see how much time I spent listening to Chappell Roan’s Bonnaroo set alone:

day	track_name	listens	length
Aug 13, 2024	Bonnaroo 2024 (Full Set)	5	291.8
Aug 14, 2024	Bonnaroo 2024 (Full Set)	2	116.7
Aug 21, 2024	Bonnaroo 2024 (Full Set)	2	116.7
Aug 30, 2024	Bonnaroo 2024 (Full Set)	2	116.7
Jul 30, 2024	Bonnaroo 2024 (Full Set)	1	58.37
Aug 01, 2024	Bonnaroo 2024 (Full Set)	1	58.37
Aug 26, 2024	Bonnaroo 2024 (Full Set)	1	58.37

That’s over 817 minutes spent listening to that set, with over a third of that listening happening on August 13, 2024.

While Spotify thinks that October 11, 2024 was a big day for music listening, as far as I can tell, it was a typical Friday at work. On the other hand, June 10, 2024 (my highest listening day according to Last.fm) was a typical Monday as well. Mysterious.

SoundCloud and my top genres

I usually compare my top genres, but Spotify didn’t even share top genres this year It’s pretty disappointing, but their genre data also does evolve over time which would have been interesting to dig into. In 2020 I noted the genres associated with Tourist and those are basically the same except that tropical house and shimmer pop are gone but future garage has been added, and electronica has been replaced with indietronica. , so instead I’ll compare my SoundCloud genres with the genres associated with my iTunes purchases. Unfortunately, Bandcamp purchases don’t contain a lot of metadata, so I don’t have any genre data for the hundreds of tracks I purchased there.

SoundCloud	Total minutes	iTunes genre	Total tracks
Electronic (Jasper Tygner)	4625	Electronic	27
Pop (Romy)	489	Dance	22
Hip Hop (Misael Deejay)	500	Pop	14
Afrobeat (Heisrema)	223	Alternative	11
Folk (Snow Patrol)	218	R&B/Soul	10

The SoundCloud playback genres really underscore what a difficult problem it is to have clean, consistent, and high quality metadata for music.

Misael Deejay was the artist listed as the representative artist for my Hip Hop listening, but I haven’t liked a single one of his tracks on SoundCloud. I was pretty confused, but then I realized that the only track of his that I listened to is the track I Have a Love (Overmono Remix) 3 times. That track is actually a remix of a track by the artist For Those I Love. This is a total bootleg track—I didn’t actually listen to Hip Hop this year.

Sorry SoundCloud, metadata is hard.

SoundCloud also took the time to share the top 5 moods of the music that I’ve listened to this year:

energetic
powerful
relaxed
quirky
exciting

Unlike the genres, these moods weren’t associated with any specific artists. I’d love to investigate further (what is quirky?!) but SoundCloud doesn’t provide API keys to their service or publish much about their research on their developer blog, Backstage, or on their official company site.

Some other mysterious moods that friends noted are “angry” and “dark”. I’ll keep brainstorming other sources I can use to dig into this further, but for now it’s tough to say where those moods are coming from.

Spotify’s “Music Evolution”

Spotify usually includes something special in Wrapped every year.

In 2023 it was streaming habits such as “Alchemist… you create playlists more than other listeners do” and a “sound town”.
In 2022, it was my listening personality (the specialist) and my “audio day”.
In 2021, it was an audio aura, with some music moods (ecstatic and innovative).
In 2020, there wasn’t a special thing, but there were stats about artist and genre discovery that haven’t resurfaced.
In 2019, Spotify wrapped up the entire decade and also shared geographic information.

This year, Spotify focused on your “Music Evolution”, calling out specific months that represented a different type of listening than you usually do. I felt like the evolutions captured three distinct patterns of listening for me this year — my pop music phase, my dance music phase, and my “oops I’m suddenly obsessed with U2’s the Joshua Tree” phase.

While I certainly had these phases of listening, they didn’t necessarily align with specific, well-separated months. My taste can shift from day to day or week to week, and doesn’t necessarily stay. For example, let’s look only at the top 4 pop artists and top 4 house artists that I listened to this year:

So where did these descriptions actually come from?

Making sense of the music evolution descriptors

Each year, I attempt to deduce the origin of and contributing factors to the “special thing” included in Wrapped. This year, I wanted to try to determine where the descriptors actually came from, and how loosely or tightly they might be associated with specific artists.

According to Spotify, I had the following musical evolutions:

Month	Descriptor	Artists
April	Pink Pilates Princess Vogue Pop	Charli XCX, Beyoncé, Chappell Roan
June	Dance Party Beats House	Fred again.., Effy, Dom Dolla
November	Apocalyptic Permanent Wave Irish Rock	U2, Snow Patrol

Lots of new-to-me phrases here (what is Permanent Wave?). I don’t really remember listening to Dom Dolla in June at all, and I think there are better artists and genres to associate with that month of listening. According to Last.fm I listened to him 6 times this year generally, and only once in June.

I probably didn’t listen to much music in June on Spotify. My top 15 artists by listens for that month were the following:

artist	listens
TC4	95
Swim	50
DJ Heartstring	32
Charli XCX	25
Bicep	20
O’Flynn	20
Barry Can’t Swim	19
Elkka	18
Jamie xx	18
dj poolboi	17
Tourist	16
Diffrent	15
Thelma	14
sjowgren	13
Caribou	12

Fred again.., Effy, and Dom Dolla don’t make the list.

As I scrolled through social media, I enjoyed the other music evolutions that I saw posted — and for the most part, I saw a bunch of different descriptors:

Many of the descriptors refer to genres, while others involve vibes or aesthetics, and still others refer to places, activities, and instruments:


Pink Pilates Princess Strut Pop	Gyaru Jam Band J-Pop
Wild West Mandolin Bluegrass	Indieheads Banjo Indie Rock
Theatrical Bandolim Choro	Pink Pilates Princess Strut Pop
Pink Pilates Princess Strut Pop	Scene Old School Emo Post-Hardcore
Breakup Banjo Folk Punk	Academic Permanent Wave Rock
Pink Pilates Princess Strut Pop	Mallgoth Skateboarding Punk
McBling Hollywood Pop	Serotonin Alt Z Indie Pop
Atmospheric Fantasy Soundtrack	Atmospheric Tavern Fantasy Music
Chill Beats Lo-Fi	Theatrical West End Broadway
Pink Pilates Princess Roller Skating Pop	Indie Sleaze Strut Pop
Pyschedelic Mid-Tempo Balearic Beat	Lounging Microtonal Neo-Psychedelic
After Hours Hollywood Pop	Academic Beats Edm
Wanderlust Metropolis Indie Pop	Pink Pilates Princess Strut Pop
Pink Pilates Princess Vogue Pop	Dance Party Beats House
Apocalyptic Permanent Wave Irish Rock	Pink Pilates Princess Catwalk Pop

Some categories that I’ve defined here can be debated (is lounging more of a vibe or an activity?) but the words mostly maintain a strong association with music. Some full descriptors double up on the genre names, like “Permanent Wave Irish Rock” or “Emo Post-Hardcore”.

Looking at this full list, however, you might have noticed a pattern…

What is up with the Pink Pilates Princess

Of the music evolutions that I saw online from friends, there were a lot of interrelated descriptors, so I compiled a list of similar ones based on what I saw on social media and that friends shared with me:

Pink Pilates Princess Strut Pop
Pink Pilates Princess Roller Skating Pop
Indie Sleaze Strut Pop
Indie Sleaze Roller Skating Pop
Pink Pilates Princess Catwalk Pop
Pink Pilates Princess Vogue Pop

The amount of overlap led me to make this venn diagram of artists associated with various descriptors:

Sabrina Carpenter has the honor of being in all 4 evolution categories, but she’s the only one that I could track down Charli XCX, Chappell Roan, and Taylor Swift nearly did too, but neither Charli nor Chappell showed up in any Indie Sleaze Roller Skating Pop examples that I found, and Taylor didn’t show up for any Pink Pilates Princess Roller Skating Pop examples that I found. Sample size is small, but nonetheless. that matched all four:

Indie Sleaze Strut Pop
Indie Sleaze Roller Skating Pop
Pink Pilates Princess Roller Skating Pop
Pink Pilates Princess Strut Pop

While it’s clear that Pop is a genre, Indie Sleaze and Pink Pilates Princess were such discrete phrases that I figured they would be derived from playlist names, probably of the artists that were represented in those categories.

I went digging into the featured playlists for each artist, and discovered the Hot Pink playlist which seems to borrow a name from a Doja Cat album and features many of the artists on the list, but it is also listed as “Made for Sarah Moir”, so it seems to be algorithmically generated as well.

Indie Sleaze is a playlist, but none of the artists in Indie Sleaze Strut Pop or Indie Sleaze Roller Skating Pop are on it!

If I search for Pink Pilates Princess, I get a hit — but it’s a Pink Pilates Princess Mix, and seems to be an AI playlist a la Spotify Premium Users Can Now Turn Any Idea Into a Personalized Playlist With AI Playlist in Beta.

Neither Strut nor Roller Skating were playlists. I thought maybe Sabrina Carpenter might have made a music video where she roller skates, but I couldn’t find any evidence that these phrases were associated in any way with the artists.

Spotify does collect cultural metadata about music, but it isn’t publicly available. It’s possible that these descriptor phrases were derived from cultural metadata that were associated with these artists in album review blog posts, TikTok tags, or social media post descriptions about the artists.

Finally, to see whether the artists themselves had similar interrelationships from a metadata perspective as their categorizations might seem to imply. With Charli XCX and Chappell Roan having the same three descriptors while Sabrina Carpenter is in all four, I wanted to dig into the related artists for each artist to determine how tightly they might be webbed together.

Unfortunately, when I went to explore this further, I discovered that the Get Artist’s Related Artists API endpoint for the Spotify API has been deprecated for “security reasons” just a few weeks ago.

As a result, the API endpoint isn’t available from the documentation and is only available for folks with an API key that is out of “development” mode (which mine is not). Introducing some changes to our Web API on the Spotify for Developers site has more details.

Where did the music evolution descriptors come from?

That was all my speculation, but the official Spotify blog in Everything You Need To Know About Your Music Evolution says exactly how the music evolution phases were identified and how the titles were crafted:

Several special factors went into creating Your Music Evolution, including the amount of time you spent listening to an artist or genre, peak listening months, and distinctiveness, which means spotlighting a genre that stood out from your usual listening patterns.

Spotify combined machine learning with human curation to associate descriptors and genres with different tracks. These descriptors then informed your musical phases.

Specifically vague. Human curation to me here means “data labeling”, with machine learning doing some generation and assignment of descriptors possibly influenced by cultural metadata. However, these word soup descriptors aren’t new to Spotify.

Ancestors of Music Evolution: daylist and Audio Day

When I first saw my music evolutions, I was like “oh this is basically just daylist”, the already unhinged playlist descriptions that Spotify has been creating since September 2023.

In Get Fresh Music Sunup to Sundown With daylist, Your Ever-Changing Spotify Playlist, Spotify introduces the daylist:

Say hello to daylist, your day in a playlist. This new, one-of-a-kind playlist on Spotify ebbs and flows with unique vibes, bringing together the niche music and microgenres you usually listen to during particular moments in the day or on specific days of the week. It updates frequently between sunup and sundown with a series of highly specific playlists made for every version of you. It’s hyper-personalized, dynamic, and playful as it reflects what you want to be listening to right now.

Given that the playlist is pitched as hyper-personalized, it’s not surprising that it’s also AI-generated.

In a NYTimes article, That Spotify Daylist That Really ‘Gets’ You? It Was Written by A.I., Frank Rojas reports:

“Spotify uses machine learning to pull together the thousands of descriptors that create the unique daylist playlist names,” Molly Holder, a senior product director at Spotify, said in a statement. She characterized the tone of the titles as “hyper-personalized, dynamic and playful.”
Ms. Holder added that the team behind these quirky playlists included data scientists and music experts who identify musical descriptors based on genre, mood and themes that are then associated with specific tracks “through methods such as music expert annotation, sonic similarity and trends.”

Put another way, the descriptors are defined by humans, and then assigned to specific tracks by data labelers and algorithms.

For reference, here’s one daylist that was generated while I was writing this post:

The music evolution and daylist descriptors made an earlier appearance in Spotify Wrapped — back in 2022, as an “Audio Day”.

According to the Everything You Need To Know About 2022 Wrapped blog post from Spotify:

Audio Day showcases the niche moods and aesthetic descriptors of the music you listened to during morning, midday, and evening time periods.

My Audio Day looked like this:

In 2022, I also investigated these descriptors. I wrote:

How did Spotify define these “niche moods and aesthetic descriptors”?

Poking around on the Spotify Research blog, I discovered the post The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood, which points out:

The association between a song and a mood descriptor was calculated using collaborative data, by “wisdom of the crowd”. More specifically, these relationships were derived from Spotify playlists’ titles and descriptions, by measuring the co-occurrence of a given song in a playlist, and the target mood descriptor in its title or description.

Researchers derived mood descriptions based on the titles and descriptions for playlists that songs were added to by Spotify users. Those mood descriptors were used as the baseline for the rest of the research, which explored whether the energy of a song and the lyrical content could be reliably correlated with mood.

And indeed, the findings of the research asserted that:

by combining information extracted from the two modalities – lyrics and acoustics – we can best predict the relationship between a song and a given mood.

I think that, based on the haphazard nature of the descriptions for My Audio Day, these were likely derived from the playlist title and description data source. Spotify is performing this research, however, to improve recommendations and search. Namely:

we want to enable search based on mood descriptors in the Spotify app, for example by allowing users to search for “happy songs”. Additionally, from the recommendations side, we want to be able to recommend new songs to users that provide similar sets of moods users might already like.

The groundwork for AI-curated mood-based playlists, mixes based on a string of words, was already in the research and Spotify Wrapped back in 2022. It’s pretty clear that this is a personalization direction that Spotify has been pursuing for years, starting (possibly) with the Audio Day, becoming widely available with daylist, then even more personalizable with AI playlists, and now the latest iteration with Music Evolution.

The backlash

I suppose that makes it somewhat surprising to see backlash to Spotify Wrapped this year, and not much over to the AI-driven personalization initiatives that have landed in the past couple years.

I remember when daylist first came out, lots of folks were sharing their daylist screenshots on Instagram and other social media sites, reveling in the weirdness (and rightness) of the descriptions.

However, with the Music Evolution this year, I haven’t seen many friends sharing screenshots of their Music Evolution on social media, and a lot more folks have shared the overall summary slide without bothering to share the Music Evolution.

Why do I think there has been backlash to Music Evolution?

It’s not very accurate

Most people don’t actually evolve their listening habits very often, let alone 3 times in one year. Some descriptors might apply to specific groups of artists, but those groups of artists might appear together on a workout playlist, or all be part of the same function (album evaluation).

I mentioned earlier how listening tends to serve a purpose, and for most people, their life doesn’t change all that much throughout the year. The exception is people who work more seasonal jobs, but even then.

In addition to whether the descriptors were accurate, the associated artists didn’t seem accurate either. I experienced this, with Dom Dolla being listed in June, and others have commented that their Wrapped seemed especially focused on popular artists, regardless of what their actual (perceived) listening habits were throughout the year.

The descriptors didn’t resonate with people

A post on Threads by jaiya (@jaiyagill) sums it up neatly:

The vibes were off —
No genres. No auras. No fun insights. It’s giving “Al-generated corporate report”.
Where’s the “OMG same” “I feel seen”conversation starters / community building moments?
It’s not just about numbers; it’s how you turn it into a story people want to share

And they’re exactly right. In previous years, the labels applied to you — you’re an alchemist, you’re a specialist, you have a specific audio aura, your audio day.

This year, I was told that my April was a “Pink Pilates Princess Vogue Pop” moment. I have no idea what that means or what I’m supposed to take away from that. Am I a Pink Pilates Princess? Is vogue pop a new genre?

It’s a mouthful, and because it isn’t grounded in personal experience, it feels rather meaningless and machine-generated.

Chatting with a friend about this, they pointed out that it would be more interesting (and personally relevant) to say something like:

you spent more time listening to pop this summer

and instead we have:

your music taste in July was after hours Hollywood pop

That type of phrasing instead “feels like the algorithm thinks I’m an algorithm and is like “hello fellow algorithm. You just discovered a new genre! I call it “soft new wave indie beats””

Dr Georgie Carroll, who wrote her PhD thesis on music fandom, elaborated on the personalization factor on LinkedIn:

Spotify Wrapped learned a hard lesson today: data isn’t enough.

Excitement built in recent weeks as fans eagerly awaited their Spotify Wrapped. It finally dropped today, and it’s been met with almost universal disappointment. Why? Because while they gave fans their stats, they took away most of the shareable fun that encouraged community and conversation. No more genres, no telling you which city your listening put you in, no auras, or any other kind of magic that made users go “omg hey, me too!”.

By cutting the human workforce that turned data into stories and relatable insights and replacing it with AI, the reactions to Wrapped are showing us why we can’t rely purely on statistics and technology (especially AI) and think it’s enough to make fans stick around. Data is only part of the story. It’s what you do with it that counts.

The AI smelled

As I mentioned, it’s a bit surprising to see poor reactions to this feature when other AI-based features haven’t yielded similar backlash. So the reactions might be less due to the use of AI itself, but which AI.

Spotify has been using AI and ML almost since its inception. The metadata, machine learning systems, and artificial intelligence have been building Discover Weekly, Release Radar, Daily Mixes, This is… artist playlists, and much much more for years — but not as much generative AI.

In an article for Business Insider republished by Yahoo News, Spotify Wrapped was a flop for some music listeners this year, a Spotify spokesperson commented:

“Wrapped is an experience that fans look forward to every year, and our approach to the data stories did not change this year,” Spotify told BI in a statement.
“We celebrated fan-favorite data stories like Your Top Artist and Top Songs with new insights like longest listening streak and top listening day,” the company said. “We’re always exploring ways to expand Wrapped and bring new data stories to users across more markets.”

Given how Music Evolution felt itself like an evolution of Audio Day and daylist, it makes sense that it was created using the same approach as previous years.

However, this year is the year of “slop” and “hallucinations” and mainstream generative AI. This is also the year that people are reacting especially poorly, associating the descriptors for Music Evolution with the sort of nonsense that a confused AI-powered chatbot might respond with, instead of interpreting it as an insightful personal response.

A feature built to represent a person’s music evolution but described using phrases that have no seeming root in something tangible really does feel like that AI slop, trying to communicate something but just ending up with word soup and melting textures.

Effect of layoffs on quality

One potential reason for the lack of accuracy, personal resonance, and AI smell of the Music Evolution feature that has been discussed online is the layoffs at Spotify last year.

On December 4, 2023, Spotify announced layoffs of 17% of the company: Spotify to cut 17% of staff in the latest round of tech layoffs.

Several months later, in their April 23, 2024 earnings call, CEO Daniel Ek acknowledged that the layoffs impacted day-to-day operations, but didn’t specify which teams or operations were affected as a result. The layoffs haven’t come up again in later earnings reports.

A viral Instagram reel by hackyourhr attributes it directly to the layoffs:

still good to meet about the 2025 budget..?

good thing a year ago to the day we laid off 1500 people, right?

the Wrapped team? AI really did its thing

so about that, umm, who’s been managing the AI that did Wrapped this year?

It just did its thing.. the AI… the AI did it

but it actually didn’t do it… we cut all those people last year to the day… no one was managing the AI… they got everything wrong…

Music journalist Zel McCarthy posted a thread on Bluesky and Threads making a similar point, writing:

If you’re a Spotify user who is finding the microgenre & geodata of your annual “Wrapped” to be less substantial than usual, there’s a reason why.

A year ago, Spotify laid off most of its employees who worked on genre classification, including folks like data alchemist glenn mcdonald.

The work Glenn and his colleagues would do—analyze vast quantities of musical content ingested on a weekly basis based on its sound, provenance, & audience then convert that analysis into data points—is no longer being done.

Yes, “Wrapped” is a marketing feature, but it presented real information.

While some of this information could theoretically be gathered by AI or other processes, it still requires human brain work.

As Glenn predicted a year ago, without him and his team, any previously automated systems would eventually desist.

And desist they have.

Personally, I have little love for Spotify’s vaunted recommendation algorithm.

In the past, I’ve written about how it reinforces biases, flattens art into product, and actually limits new music discovery while purporting to do the opposite.

Still, depriving the algo of data only makes it worse.

tl;dr your #SpotifyWrapped is lamer this year b/c the tech company where you get your music has divested from music-based research and analytics.

The music world has less information and has lost the means of gathering it in the future… and as of now, there’s nothing we can do to get it back.

I haven’t been able to independently confirm that Spotify laid off most of the folks that perform genre classification (the data labelers I mentioned earlier, in part). Since Zel McCarthy is a journalist, I trust his reporting here, but I still dug around on LinkedIn.

There are still folks that work on Spotify Wrapped, but a number of data scientists and data curators that previously worked on Wrapped — including someone who described their work as being the human-in-the-loop for data curation, sourcing and evaluating data and interpreting datasets — have been laid off.

Others chose to leave. Ajay Kalia (who was a former Echo Nest employee along with glenn mcdonald) left around the same time as the layoffs occurred to start his own business. Kalia was the Product Director for Personalized Experiences, which was responsible for launching Daylist, among other personalized playlist and other initiatives. Molly Holder, who was quoted in the New York Times article about daylist, remains at Spotify working on personalization.

Even without knowing more specifics about who got laid off, the 2024 version of Spotify is focused on being “relentlessly resourceful” and laying off 1500+ people in order to “rightsize” as a noun. 2021, 2022, and 2023 are years when Spotify did wild things like pay someone to read auras and take you to sound town.

This year’s Wrapped shows that a viral marketing campaign isn’t a “set it and forget it” exercise. The data is the interesting part of the insights, but they need to feel relevant and personal. That doesn’t happen with a word salad of insights, and it does seem to require talented people.

Wrapping up…

Spotify Wrapped is always going to be a marketing exercise first. If you want to truly understand your accurate data, you need to take the quantified self approach of tracking listening activity on Last.fm, or downloading your listening data directly from Spotify and exploring it.

Overall, one consistent point throughout this post is that data isn’t necessarily personalized, but it is personal. Metrics are one layer of a broader picture, but metrics also are what Spotify has built its business on.

Building datasets of listening behavior, profiles of consumers, and packaging that up into advertising verticals serves as the primary engine of Spotify’s revenue. Building metadatasets with algorithmic and human intervention is the foundation of the sticky playlists and mixes, like Discover Weekly, Release Radar, daylist, and more.

But most quantitative metrics are derived metrics. Even something that seems easy to measure, beats per minute (BPM), is derived, and not that accurate all the time.

Despite tracking my listening behavior and habits, I still make a separate list of what my favorite tracks of the year are, which artists I discovered that are new favorites, and identify my favorite DJ sets. I can assess the frequency, consistency, and popularity of what I listen to for days (and I have, writing this post), but that’s not the same thing.

The true personalization is what music means to me, and that can’t be algorithmically derived.

Domain ownership of ccTLDs and sovereignty

Tue, 08 Oct 2024 00:05:43 +0000

Tianyu Fang researched a long and detailed essay called Whose Domain Is It?, detailing the politics of the Internet’s domain names, especially the country-code top-level domains (ccTLDs) that you might not realize are actually associated with countries.

The unexpectedly popular ccTLDs can operate like a tourism effort, at least in terms of generating unexpected revenue for small countries:

Domain sales are generating revenue for Anguilla’s government. Per Cate’s estimate, the domain registry is currently generating $3 million in revenue every month for the government, which is somewhere around a third of its monthly budget.

Anguilla isn’t the only microstate with significant government revenue from domain sales, though its relative scale is unmatched. Tuvalu, an island nation in the South Pacific, famously paid for its entry to the United Nations by selling the license for .tv to a US firm at the height of the dot-com bubble; its government made close to $5 million from ccTLD sales in 2022, or 8% of its total revenue that year. Montenegro’s revenue from the .me domain amounted to 2% of total exports in 2015, with the vast majority of the registrant coming from abroad.

These top-level domains are sold, traded, and auctioned as commodities—to be exported by small governments to foreign startups, registrars, and investors in exchange for revenue. But it hasn’t always been that way: in fact, the internet’s early pioneers had intended these digital resources to be community-run infrastructure that served a public function.

Of course, the registrars managing the ccTLDs might not actually have any ties to the government. When ccTLDs were introduced, the registrars weren’t necessary allocated or assigned in an exceptionally precise way. It was essentially first-come first serve:

In 1994, the California native and Berkeley libertarian Vince Cate dropped out from his PhD program at Carnegie Mellon, where he was designing the Alex filesystem. He wanted to relocate to a tax haven but couldn’t afford living in Bermuda or the Cayman Islands. He moved instead to Anguilla, where he paid $470 a month on rent to kickstart an email business. In Anguilla, he was the first to email Postel, who told him that there wasn’t anyone managing Anguilla’s TLD yet and suggested he be the first volunteer manager of .ai.

In this case, the registrar managing the .ai ccTLD is at least a resident, but that’s not always the case:

Many registries, such as .us and .fr, introduced regulations to limit domain registrations to individuals or organizations with actual ties to the country, though they’re enforced to varying degrees. This process of formalization—the transfer of administrative power from volunteers to governments—was complete in most major countries by the early 2000s. In Anguilla, too, Cate changed the administrative contact of .ai to the government of Anguilla, which gladly kept him as the technical manager.

A ccTLD might not be as stable as a generic TLD (gTLD) or as you might think something based on a country might be. Country borders and sovereignty can evolve, especially for territories subject to decades of imperial rule. For example, the .io ccTLD might end if and when British occupation of the Chagos Islands ends:

It’s possible that the British Indian Ocean Territory won’t exist one day—international courts have ruled against British occupation of the Chagos Islands, and the UN General Assembly called for the archipelago’s decolonization in 2019. If these names are struck off the ISO list of countries, we might be left with a world of dead links. The loss of domain names will be trivial compared to the real-world implications of lost lands and lives, yet the cultural heritage that comes with it will, too, be consequential.

The entire essay is fantastic and I recommend reading the entire thing. For more on the current state of the .io ccTLD, see When imperialism ends, so too might the popular .io ccTLD.

WHOIS vulnerabilities and TLDs

Mon, 07 Oct 2024 23:43:30 +0000

Most of the Internet is held together by best practices and good intentions, and WHOIS servers are one of those. One security company was investigating vulnerabilities in WHOIS and got a whole lot more than they bargained for:

Each TLD (the bit at the end of the domain), you see, has a separate WHOIS server, and there’s no real standard to locating them - the only ‘real’ method being examining a textual list published by IANA. This list denotes the hostname of a server for each TLD, which is where WHOIS queries should be directed.

As you can imagine, maintainers of WHOIS tooling are reluctant to scrape such a textual list at runtime, and so it has become the norm to simply hardcode server addresses, populating them at development time by referring to IANA’s list manually. Since the WHOIS server addresses change so infrequently, this is usually an acceptable solution.

However, it falls down in an ungraceful manner when server addresses change.

IANA is the Internet Assigned Numbers Authority responsible for managing the DNS root zone, including top-level domain names like .mobi.

When imperialism ends, so too might the popular .io ccTLD

Mon, 07 Oct 2024 23:42:07 +0000

The UK will give sovereignty of Chagos Islands to Mauritius:

Half a century or more after the UK relinquished control over almost all its global empire, it has finally agreed to hand over one of the very last pieces. It has done so reluctantly, perhaps, but also peacefully and legally.

The Chagos Islands are also known as the British Indian Ocean Territory, to which the .io ccTLD is assigned. Given that the UK is relinquishing control of the islands to Mauritius, there will no longer be a British Indian Ocean Territory, raising questions about the popular .io ccTLD.

Simon Willison summarizes the Hacker News discussion on his weblog:

It sounds like there’s a very real possibility that .io could be deleted after a few years notice - it’s happened before, for ccTLDs such as .zr for Zaire (which renamed to Democratic Republic of the Congo in 1997, with .zr withdrawn in 2001) and .cs for Czechoslovakia, withdrawn in 1995.

The Wikipedia article for the ccTLD is also informative.

AI and a duty of care

Fri, 12 Jul 2024 23:09:09 +0000

A recent post from the American Alpine Club has me considering the potential duty of care that websites like AllTrails and Mountain Project owe to their readers.

In The Prescription—July 2024, the editor Pete Takeda describes two incidents, one where two people were stranded due to their inexperience climbing snow, and another where one person in a group of nine slipped and fell to her death.

Recent accidents in Grand Teton National Park

The American Alpine Club Prescription highlights a route on the Teewinot mountain in Grand Teton National Park that has claimed many lives. Introducing the accidents, Takeda points out:

While these accidents differ in immediate cause and final outcome, they share a common origin: the use of hiking-specific applications for selection, preparation, and route-finding, versus the use of climbing-specific resources.

In the conclusion and analysis, he goes further, pointing out the use of hiking-focused apps as a key root cause:

Use of Hiking Apps. This team relied on information taken from a popular hiking application (as previously discussed in this Prescription). This climb in particular is listed on several platforms as a hike. Interviews with survivors revealed they were under the impression that the East Face of Teewinot was a traditional hike. It is a fifth-class climb.

Based on this information, he researched the AllTrails description of the route:

On the app, the updated route description reads like a ChatGPT-derived synthesis of crowd-sourced user comments. What it lacks in human nuance, it compensates with a tinny stridency.

To me, that immediately makes me wonder — is it actually generated text?

Is it generated?

When you open the AllTrails page for Teewinot, the route is described as a mountaineering route, but the description sounds largely innocuous:

Proceed cautiously on this 5.9-mile out-and-back trail near Moose, Wyoming. Generally considered a highly challenging route, it should only be attempted by experienced adventurers. This trail is great for rock climbing, and it’s unlikely you’ll encounter many other people while exploring. The best times to visit this trail are July through September.

As Takeda also highlights, referring to this route as a trail in the summary is extremely misleading, and the text is almost certainly AI generated. How can I be so confident?

A press release published by AllTrails on June 13, 2024, titled AllTrails Reveals Major Update advertises Redesigned Trail Pages:

Redesigned Trail Pages

AllTrails overhauled its more than 400,000 Trail Pages to help trailgoers plan with confidence, making essential information more accessible and visually engaging. Key details like trail length, format, and estimated duration are more prominent, and the pages now include the ability to visually tour the trail in a dynamic Photo Gallery, view the route in 3D, assess trail conditions, and read an AI-powered summary of community reviews.

However, there’s nothing on the page to indicate that the summary is AI-generated, nor any details about what prompt is being used to shape the community reviews into a summary of the route.

A local route near me that I discovered on AllTrails contains similar language:

Enjoy this 59.5-mile out-and-back trail near San Francisco, California. Generally considered a challenging route. This trail is great for road biking, and it’s unlikely you’ll encounter many other people while exploring.

As someone who spends a lot of time crafting consistent content, this type of structured description is unsurprising. What is surprising is the use of the word trail for routes that a person wouldn’t describe that way.

Using the term trail to refer to a mountaineering route or a cycling route indicates to me that this text is almost certainly machine-generated, and that the instructions include guidance to use that term.

Data cleanliness and prioritization

The use of the word trail to describe these non-hiking adventure routes isn’t a matter of data cleanliness. The fact that these routes aren’t for hiking and are instead for climbing or cycling is known, but not clearly communicated.

It’s an issue with how the data is prioritized and communicated on the site.

If you filter for hiking trails in Grand Teton National Park, this route does not appear. However, if you filter by the activity “Rock Climbing” this route does show up. The data is there.

However, on a trail description page, there is no “activity” filter or label — only a loose set of barely noticeable tags or labels that describe the route:

If this were an issue of unclear or missing data, filtering by activity wouldn’t be possible at all. Instead, the data is there but isn’t clearly highlighted or communicated on the page.

Don’t try to be the tool for everything

A platform can be immensely valuable in a specific niche. AllTrails is fantastic for offering granular route details, community-sourced conditions, and other relevant information for hiking trails.

But AllTrails was built for hiking trails, and the machine generation tooling used to create summaries was clearly also built for hiking trails—despite the site now including a variety of routes for other activities.

The expertise and audience that built AllTrails into what it is today for hiking doesn’t automatically translate to other activities like rock climbing and cycling, and as such, isn’t appropriate to offer support for rock climbers (and probably cyclists too) the way that dedicated sites can.

Mountain Project, the go-to source for most online climbing information, is part of a conglomerate that has expanded over time to include mountain biking, hiking, trail running, and skiing. And it’s telling that the sites are separate — because the audiences and the risks are distinct.

Expanding to a new customer base requires building intentional content, and in the outdoors, access to that content often requires local community experts. Other sites that serve climbers and mountaineers, like SummitPost, or the climbing app Kaya, offer support for community experts and are mindful stewards of information.

It might not seem like a big deal to generate summaries for the different routes listed in AllTrails. It might even seem like a net benefit for consistency and clarity, making the outdoors more accessible and available.

But if those summaries are crafted by a poorly-instructed machine, rather than a person that understands the context and the risk involved in providing such a summary, you end up with a mountaineering route called a trail, and a 60 miles cycling route called a trail.

A product with “AI-powered summaries” that are actively misleading is a product that is neglecting its duty of care to the outdoor enthusiasts it aims to support.

A career bucket list for technical writers

Fri, 19 Apr 2024 08:53:07 +0000

What’s next for you in your career? It’s tempting to focus on the bare minimum — staying employed — but identifying new areas of professional development or focus can help you grow your career and find whatever enjoyment you can from the capitalist toil that is an obligation of modern life.

While I was at Splunk, Susan St. Ledger gave a talk about approaching her career with a bucket list. I’d defined my career values, but a bucket list for my career was the perfect way to complement my career values while still growing my expertise.

Even if you have the same title, you can have different experiences with your career and learn different skillsets if you seek out different team sizes, company stages, reporting structures, and working environments.

Identifying that variation has been crucial for me as I’ve stared down a lifelong career doing “just writing”. Did I really want that? Thankfully, technical writing doesn’t look the same everywhere, and that variation is what keeps it exciting for me.

In chatting about next steps in a technical writing career with a mentee, we came up with the following list of experiences and job situations that could be on a bucket list for technical writers.

What types of experiences and job situations might make sense on a bucket list look like for technical writers? I’m still developing my own, but I wrote up this list to serve as inspiration:

Tooling:

Migrate a doc set to a new platform or tool.
Implement a doc feedback collection tool.
Implement an in-product tutorial tool, such as WalkMe, Pendo, Gainsight, or similar.
Maintain and customize a documentation platform.

Writing process:

Write in a docs-as-code workflow.
Write structured content like with Doxygen or DITA-based tools.
Write in a different markup language (HTML, MediaWiki, Markdown, Restructured Text, AsciiDoc, DITA XML, etc.).

Writing tasks:

Write a documentation set from scratch.
Write API documentation.
Write technical content marketing and/or blog posts.
Write in-product tutorials and tours.
Write tutorials.
Write stellar reference content.
Reorganize or rearchitect a documentation set.
Write UI text and error messages.

Writing and design tasks:

Create detailed diagrams.
Design graphics and icons.

Collaboration or stretch opportunities:

Collaborate with marketing on content.
Do product development, such as by writing an interactive in-product tutorial, or committing code to the product.
Design a documentation site.
Perform user research.
Do customer support.
Work with developer relations/advocacy on content.
Develop best practices documentation with the field (sales engineers, professional services consultants, etc.).
Partner with the web team on SEO improvements.
Start a naming council to codify and manage product and feature names.

Writing standards:

Write a documentation style guide.
Write a UI text/content design style guide.
Define audiences or personas for the docs.
Work with editors.
Define templates for reference and other content.

Writing strategy:

Define documentation strategy and vision for a product/company.
Make a business case for documentation.
Develop documentation analytics KPIs and success metrics.
Define and implement a triage process.
Formalize and mature documentation planning processes.

Personal brand:

Present a talk at a conference like STC Summit, IAC, Write the Docs, LavaCon, Button or others.
Speak on a podcast.
Participate in a panel discussion.
Write about documentation for the company blog.

Writing-relevant roles:

Be a manager of a large team.
Be a manager who grows and scales a small team.
Be a team lead.
Be a lone writer.
Be a minion at a large company.
Mentor a teammate.
Become an editor.
Be a scrum master.
Be a content designer.

If you’re looking for what’s next, whether at your current company or somewhere new, I hope this list helps you consider what you might want to accomplish next!

Docs as code is a broken promise

Wed, 10 Apr 2024 21:27:12 +0000

Docs as code is a much-vaunted workflow and toolchain for writing, publishing, and maintaining technical documentation — but in practice, docs as code doesn’t deliver on its promise.

What is docs as code?

According to the Docs as Code page in the Documentation guide for Write the Docs:

Documentation as Code (Docs as Code) refers to a philosophy that you should be writing documentation with the same tools as code:

Issue Trackers

Version Control (Git)

Plain Text Markup (Markdown, reStructuredText, Asciidoc)

Code Reviews

Automated Tests

This means following the same workflows as development teams, and being integrated in the product team. It enables a culture where writers and developers both feel ownership of documentation, and work together to make it as good as possible.

This is promising — shared ownership of documentation and a shared goal to make documentation as good as possible by sharing the same tools and processes.

All too often, however, I find that this philosophy is adopted as a set of tools, and the processes and integration are ignored. Even if you have processes in place, the tools you use to do docs as code make it easy to circumvent the processes, even unintentionally.

The promise of docs as code

If your documentation workflows follow the development workflows, as a technical writer, you can get your job done more easily.

If you do docs as code…

Documentation reviews can take the same format as code reviews—a pull request—making it easier to get reviews from engineers.
Code samples can be automatically tested, or better yet, pulled in at build time from the product code.
Doc changes can happen in sync with the product changes if docs are in the same repo, streamlining the release cycle.
Documentation can be versioned, and you can keep track of when and how a given page was last updated.

The pitfalls of docs as code

Writing content isn’t the same as developing code, and as a result, doing docs as code has pitfalls:

Let me explain…

Git is confusing

To do docs as code, writers need to learn how to use and troubleshoot Git. And Git isn’t simple — Julia Evans made an entire Zine about it and has been working on a longer series about Git concepts to make it much clearer.

There is a deep level of complexity with Git, and while you can go a long way only knowing how to git fetch, git pull, and git merge, it’s easy to get into an unexpected state with your cloned Git repo, local branch, or disastrous merge conflict resolution decisions. In those cases, often the best troubleshooting is starting over — and that’s not a great experience!

Processes must be defined and reflected in Git workflows

When you use a docs as code workflow, you need to codify your docs processes and instantiate them in your Git workflow. So you not only need to define the following:

When to publish doc updates
How to release doc updates
How to coordinate a docs release with a product release

You also need to define Git best practices for your team about how to manage those, such as:

Whether to use release branches, or merge pull requests frequently but publishing infrequently.
Whether to use Git rebase or Git merge to maintain Git history in a given branch.
Whether and how to use feature branches and pull requests.
Whether to squash merge pull requests to main.

Even if you manage to define best practices that your team is committed to following, there isn’t a way to force your documentation contributors to adhere to all of these best practices. Due to the lack of enforcement of these best practices, you can easily end up in a situation where writers follow slightly different practices based on what their tools make easy to do.

As just one example, I spent several weeks merging all my pull requests to main, instead of squash merging as was the best practice. At some point, I’d needed to do a regular merge and preserve all my branch commits on main, and GitHub “helpfully” remembered my last-used setting.

Because the best practice was just a practice, and not enforced, it took me several weeks of cluttering up main with my chaotic branch commit history before I realized what was happening. In the meantime, the best practice of squash merging, meant to maintain a clear association between a PR of feature changes and a Jira ticket, was completely ignored with no warning.

Tools to write docs can be inconsistent

In a typical docs as code environment, you write locally and push your changes up to a central repository. Unlike a typical content management system (CMS) like WordPress or Drupal, or software like MadCap Flare, the writing environment isn’t shared — only the content is the same.

Instead, the writing environment is relatively uncontrolled by default — a writer can choose to write their content in whatever system they want, so long as they can commit text to the Git repository.

A writer contributing to the documentation can write in a basic text editor like Sublime Text, an extension-filled Visual Studio Code setup, in vim, or some other tool — even the GitHub or GitLab UI!

Because only the end result matters to docs as code, different writers and documentation contributors can use different tools with different settings and functionality. However, these environment inconsistencies can lead to inconsistent content quality and style, and make issues with content development more difficult to troubleshoot.

If you use a text editor that doesn’t have a spell checker installed, you can easily introduce typos into the documentation — but only the topics that you contribute to. If your colleague doesn’t use a system supported by the automated style checker recommended by your team, the style of their documentation topics will deviate from company style.

Merge gates and build checks are great… if you have them

One advantage of docs as code is the extensibility and automated nature of the workflows. Because you can treat documentation content like code, you can implement some code-like practices, like checking the validity of your documentation markup when you propose changes to the documentation with your pull request (a merge gate) or when you build the documentation site (a build check).

However, to take advantage of these capabilities, you often need to build the tools and checks yourself, or get your documentation platform team to build them for you — right after they work on the rest of the backlog of improvements for the customer-facing documentation site.

Some companies put a lot of effort into engineering efficiency, with entire developer experience teams devoted to improving engineering workflows. Meanwhile, documentation teams are often lucky to get one engineer to work on the documentation site itself, let alone help enable any special docs as code workflows. This makes sense for the business, but it makes it difficult to take advantage of what docs as code can do for you.

For example, you could set up merge gates or build checks to perform documentation quality improvement tasks like the following:

Optimize image size
Lint content to standardize and correct markup
Check for broken links
Test code examples
Validate adherence to the style guide
Check spelling and grammar

Some of these might make more sense to implement in your writing tool directly (such as with Visual Studio Code extensions), but then you would need to require writers to all use the same writing tools configured in the same way.

Without resources to implement custom merge gates or build checks for documentation quality, documentation writers have the added process burden of opening a pull request, requesting reviews and approvals, and miss out on the potential advantage of automating tedious tasks. And if their content is in the same repository as the product code, they also need to wait for a bunch of unrelated tests to pass before they can merge any doc updates.

Reviewing content in the repository is hard to read

A much-touted benefit to docs as code is that the content is in the same tools used by engineers, which makes it easy to get technical reviews done in GitHub, GitLab, or your Git provider of choice.

Unfortunately, documentation reviews in a pull request can be confusing. Just like reviewing UI code is difficult if you only have access to the source and not a staging environment, if you can’t provide your reviewers with a staged or preview version of the content, you might get a technical review full of confused comments instead of helpful feedback.

An engineer might see that you deleted lines from a file, not realizing that you deleted them because they were in the wrong place, not because the information was wrong.

Someone else might comment that you need to provide a note on several pages, not realizing that you did that — using a single sourced content reference that isn’t obvious to someone who doesn’t write documentation.

Parsing a documentation pull request and providing helpful comments requires the reviewer experience the content the same way that a future customer will — but a pull request only provides the raw, marked up text.

Upholding the docs as code promise

In its current state, with these pitfalls I’ve outlined (and more), I find it problematic to recommend docs as code as a one-size-fits-all solution for all your technical writing CMS needs. Docs as code is a workflow with processes and tools, and therefore requires investment, maintenance, and a decent amount of custom tooling to get true value out of it.

It’s possible that using a well-resourced static site generator like Docusaurus or MkDocs as part of your docs as code workflow might address some of these pitfalls, but from my impressions writing, building, and deploying this blog using the Hugo static site generator, it’s unlikely. I’d guess that those give you extensions and flexibility to make a static site work well for documentation, but they don’t solve the core writing experience challenges of a docs as code workflow.

If you’re considering implementing docs as code workflows, you can choose the degree to which you want to adopt the practices. If you require a branch, pull request, and all build checks to pass to fix a typo, the time it takes to fix a typo could easily triple. On the other hand, with that level of overhead, you gain ultimate auditability of documentation changes — the what, when, why, and by whom.

When considering docs as code workflows today, I often see what my friend referred to as the “most extreme option” — a writing team and toolchain that has adopted all the practices from code development. I think to uphold the promise of docs as code, we could do to add a bit more of the docs back into the workflow.

As Fabrizio Ferri Benedetti put it in his post, The pros and cons of using Markdown ¹:

you want the docs to be the product […] But be very careful about the product not being, for example, the pipelines or the site you’re gonna render.

There are newer CMSes on the market, like ReadMe, Heretto, Paligo, and others. From my extremely limited experience using one of these CMSes, it seems like the writing experience is much simpler, and there is a way to write Markdown without needing to use developer level tools.

Unfortunately, it seems like these tools also still need to mature and borrow more of the practices that do work in docs as code, like being able to compartmentalize documentation drafts and release specific changes at a specific time.

It could be as well that we are early enough in the docs as code ecosystem that the tooling hasn’t yet matured—but Anne Gentle’s book Docs Like Code came out in 2017, so this concept has been around for some time.

I sincerely hope that the content management systems improve, or that we see more standardized toolchains with guardrails that enforce best practices for doing docs as code.

The easier it is to implement docs as code — consistently, without much room for individual writer error — the easier it is to focus on producing high quality documentation, together. This is exactly why large companies have entire engineering efficiency teams.

What would it take to get a writer efficiency community to focus on a spectacular docs as code experience?

That post is a follow-up of his comments in the webinar, Pros and Cons of Using Markdown for Technical Documentation Panel Discussion, which covers some of the same ground as I do in this post (and beyond), such as things about availability and scalability of the content for developers contributing to the content, and an illusion that implementing such a workflow can be free. ↩︎

What about GIFs instead of screenshots?

Sat, 06 Jan 2024 18:16:57 +0000

After I published Should you add screenshots to documentation?, I got some comments from folks who prefer GIFs to screenshots because GIFs can more clearly show how to use a complicated user interface.

I agree that GIFs are cool and useful, but they’re also MUCH harder to keep up-to-date than screenshots and have extra accessibility considerations if you decide to use them.

WCAG level A standard requires that:

For any moving, blinking or scrolling information that (1) starts automatically, (2) lasts more than five seconds, and (3) is presented in parallel with other content, there is a mechanism for the user to pause, stop, or hide it

If you use a GIF, you need to consider those accessibility constraints, as well as the standard consideration when providing visual task steps: Can someone complete the task successfully if the screenshot, GIF, or video is unavailable?

So if you decide to use GIFs instead of screenshots, keep in mind these extra considerations:

Can someone pause, stop, or hide the GIF?
Does the GIF stop moving in less than 5 seconds?
Did I write effective alt text for the GIF?
Is the source file for the GIF stored somewhere accessible to the rest of my team?
Do we have clear and consistent style guidelines for creating GIFs?

Wrapping up my 2023 in music

Sat, 30 Dec 2023 21:25:27 +0000

2023 has been a weird year in music for me! I discovered a lot of new-to-me corners of music, didn’t go to very many shows by my standards, and largely flitted between new and familiar artists all year. Let’s dig in…

Each section of this post is pretty standalone, so feel free to skip around:

Top artists of 2023

My top artists of the year were, like in years past, a mix of familiar and new artists:

click to enlarge

Bombay Bicycle Club absolutely dominated my listening activity. This year they were a real comfort listen for me, and they released a new album. They also released a live album last year that really got me missing their live shows, so I was overjoyed to see that they’re back touring North America next year.

While Bombay Bicycle Club were a fairly consistent listen throughout the year, I devoted a good amount of listening time to other artists too:

In terms of familiarity and newness, 3 of my top 10 artists are artists that I discovered over 10 years ago, while I discovered the rest in the past 3 years.

artist	year discovered
Bombay Bicycle Club	2009
Barry Can’t Swim	2020
Overmono	2020
Hey Rosetta!	2010
Fred again..	2020
Sofia Kourtesis	2020
TSHA	2019
Romy	2020
Bon Iver	2009
DJ Seinfeld	2019

What do these artists have in common that might make them some of my most-listened-to artists of this year?

Hey Rosetta! and Bon Iver

Most of my time listening to these artists was in short-lived but intense bursts, concentrated in days or nights where I was trying to get something done or calm down. I sought out familiar, calming music like Hey Rosetta! and Bon Iver.

Any artist that I discovered during college could fit this bill well, but some are better served for wallowing (Cold War Kids, The National) than relaxing.

Barry Can’t Swim

When Barry Can’t Swim released his single Sunsleeper, it was so catchy. Sunsleeper ended up one of my most consistently listened to songs of the year, and my interest in his album spiked. I started eagerly awaiting his album release—so eagerly, in fact, that I accidentally pre-ordered his album twice.

The album came out shortly before I took a cross-country flight and I listened to the album on repeat for at least half the flight. Good stuff, and a consistent listen throughout the year. I also saw him live in March, which helped stoke the interest fires. As his handler for the night ushered him past the crowd, he asked “How did you discover him?!” as though awestruck by the fact that he’d sold out the smallish club venue.

Overmono

Overmono has been so consistently great that I almost didn’t remember that their album, Good Lies, came out this year. Their live set at 1015 Folsom in April was even better than their set at Monarch last year, and their Manchester Boiler Room is entrancing. I’ve watched it at least 3 times.

Sofia Kourtesis

I heard a couple songs of hers in 2021 (notably La Perla) and checked out her set at Portola in 2022 briefly, but when I saw she was coming out with an album this year I knew I had to pre-order it. And it was great! One of those albums where no one song burrows in your mind, but the whole album is enjoyable.

Romy

It took me some time to lean into this album. Most of my listening was me putting it on to listen through, realizing that it had ended without really gripping me, and then hitting play again from the beginning.

With the repeated listens, and the tracks that I didn’t already know as singles eventually sticking with me, I recognized the album for what it is — purely enjoyable and hella queer pop tracks.

Her London Boiler Room set really captured me though, especially when she almost immediately cut to one of my favorite pop songs of all time, Ariana Grande’s Into You.

DJ Seinfeld

I wasn’t expecting to see DJ Seinfeld on this list. I ranked his 2021 album Mirrors as one of my favorites of the year, but aside from discovering him that year, he barely showed up in my data.

What happened this year is that I pivoted the vast majority of my listening to SoundCloud. It was an intentional decision, but definitely facilitated by my company’s strong IT policy that doesn’t allow unapproved apps like Spotify to be installed, limiting me to web-based listening. Spotify web is not great.

So as a result, I was listening to SoundCloud Weekly every week instead of Spotify’s Discover Weekly playlist. And it’s a great playlist, but it doesn’t quite hit the blend of new to familiar that I might hope for. That means I end up listening to a bunch of familiar artists repeatedly — TSHA, DJ Seinfeld, Amtrac, and Fred again.. — that I might not have sought out as regularly otherwise.

Not mad, just surprised!

My SoundCloud 2023 Playback playlist is pretty indicative of both the repeat listens from SoundCloud Weekly, and the artists that I played and luxuriated in the high quality shuffle discovery that SoundCloud offers.

Some low-key obsessions

Those were my top 10 artists of the year, but some other new music that I discovered this year really struck a chord (lol) with me and led to some noticeable-in-the-data obsessions.

DJ Heartstring

A friend shared the DJ Heartstring Boiler Room the day it came out, and I was an immediate fan. I’d never heard of them before, but I listened to that set a bunch of times and then spent the next few months tracking down the unreleased tracks they played as they started to release them.

As a result, my Last.fm genre patterns look like this:

Techno shows up in May sometime and holds strong all the way until September.

I don’t have genre data by month for Spotify or iTunes, but there are some similar patterns:

Pop music

The theme across all of these genre charts that I haven’t mentioned yet is pop music. I eschewed pop music for most of my teen years and have some friends in freshman year of college to thank for introducing me to artists like Pitbull and Jason Derulo and teaching me to like pop music.

Since then, I’ve largely stuck to artists like Ariana Grande or Carly Rae Jepsen. This past year, inspired by an occasional show on KCHUNG radio and a collaborative playlist with two of my best friends, I dove deep into what I’ve been calling “modern pop” from artists like Ava Max, MUNA, Dagny, Caroline Polachek, and others.

Those deep dives reveal themselves as mini obsessions in my music listening habits and my Spotify Wrapped.

The top 15 or so artists from my Spotify Top 100 songs of 2023 playlist reflect this interest:

Artists like Mallrat, Griff, Abby Sage, Arlo Parks, Gatlin, Ava Max, ELIO, and HAIM show up strongly in my Spotify listening habits, but are much less represented in my overall top listens of the year:

By contrast, only Dagny, Abby Sage, Griff, and ELIO show up in that chart — and those are the artists whose tracks I purchased. I also own some Ava Max tracks, but my listens of her peaked in January and petered off by March, while my listens of ELIO and related artists was spiking around March and April along with most of my pop listening, as the Last.fm genre chart also reflects, and was more widely dispersed over the rest of the music.

click to enlarge

Looking at just the first third of the year, Bombay Bicycle Club and Overmono consume a lot of listening time, but ELIO and Ava Max still contribute.

Largely, these pop songs are much more indie-influenced — they’re emotional and moody, yet catchy. That seems to describe most of what I’ve been listening to lately tbh.

Genre explorations — what is Alt Z?

Spotify Wrapped 2023 included a nice genre sandwich:

I love to see Future Garage on there, and Pop and Indie Pop are unsurprising as previously discussed, while Indietronica is in my top 5 every year. But Alt Z? That feels totally made up.

I’ve gone into detail about the complexities and hyper-specificity of Spotify’s genre categorizations before, but this still felt like a lot.

I figured this genre likely came from one of my top artists according to Spotify, and it turned out I was right.

Using Spotify’s Get Artist API endpoint, I retrieved the genres of my top 5 artists, and ELIO, the artist that I listened to second-most this year according to Spotify, had the following genres:

[
  "alt z",
  "electropop"
]

That’s one mystery solved. But what is Alt Z? I dug a little deeper by calling the Get Artist’s Related Artists endpoint to see which artists were related to ELIO, and if any of them had the same Alt Z genre.

Turns out that a lot of them do:

    {
      "genres": [
        "alt z",
        "bedroom pop"
      ],
      "name": "Sophie Cates",
      "popularity": 33,
      "type": "artist",
      "uri": "spotify:artist:4xjJOu0MWVWuaDVZOy0Dx2"
    },
    {
      "genres": [
        "alt z"
      ],
      "name": "Julia Wolf",
      "popularity": 44,
      "type": "artist",
      "uri": "spotify:artist:5yvGiZLSWJTPBlZpVbPnEZ"
    },
    {
      "genres": [
        "alt z",
        "indie pop",
        "nyc pop"
      ],
      "name": "Maude Latour",
      "popularity": 44,
      "type": "artist",
      "uri": "spotify:artist:3MNLhvqJkWsO6tcjY9ps62"
    },
    {
      "genres": [
        "alt z",
        "indie pop",
        "indie poptimism",
        "uk pop"
      ],
      "name": "The Aces",
      "popularity": 49,
      "type": "artist",
      "uri": "spotify:artist:2AmfMGi3WZMxqFDHissIAe"
    },
    {
      "genres": [
        "alt z",
        "nyc pop"
      ],
      "name": "Heather Sommer",
      "popularity": 46,
      "type": "artist",
      "uri": "spotify:artist:0EHYuPn9Xng2lZP2cfz4cV"
    },
    {
      "genres": [
        "alt z",
        "canadian pop"
      ],
      "name": "renforshort",
      "popularity": 42,
      "type": "artist",
      "uri": "spotify:artist:3GYvf7puxwkr51EYoD9E7D"
    }

In listening to the artists, it seems like the things in common are that the artists are all women and somewhat young. To my ear at least, it’s difficult to hear how Julia Wolf, with echoes of mumble rap, might be the same genre as The Aces, with echoes of Alvvays, or as Sophie Cates, with echoes of Stars’ track Calendar Girl.

Given that the genre is called Alt Z, I figured it was likely a reference to Generation Z, so I looked up the ages of the artists listed here. Sophie Cates is 24, ELIO is also 24 or 25, Julia Wolf is 28, Maude Latour is 24, Heather Sommer is 28, renforshort is 21, and while I couldn’t find exact ages for the women in The Aces, they seem to be in the same age range. Most of these ages are on the older side for Gen Z, so I don’t think it’s a valuable characterization.

Perhaps people that Spotify thinks are part of Generation Z listen to these artists, though I don’t think an “Alt Millennial” genre would be coherent at all, given the relative meaningless of generational labels for drawing conclusions. Pew Research Center, the preeminent population research centers in the United States, recently stopped using generational labels when doing research because it often lends itself to lazy research methods. Instead, they are shifting their focus: ‘When comparing generations, it’s crucial to control for age. In other words, researchers need to look at each generation or age cohort at a similar point in the life cycle.’

Even if I check out the Alt Z genre map on Every Noise at Once to see all the artists that Spotify lists in that genre, I’m still hard-pressed to identify a common theme across the artists.

What are the Spotify streaming habits?

Every year, Spotify tries to imbue Wrapped with some ~ * insights * ~ and this year was the most boring in awhile, in my opinion. Spotify’s blog post Me in 2023 Reveals the Streaming Habit That Defined Your Listening describes the different streaming habits that characterized various types of listening this year on Spotify.

Perhaps most interesting to me is that their blog post lists 12 different streaming habits, but when I asked my friends that use Spotify, those who replied to my question only ended up with one of five different characterizations:

Alchemist: Listening is your laboratory. You create playlists more than other listeners do. Nice work, doc.

Hypnotist: Your concentration is absolute, friend. You like to play albums all the way through, from the opening track to the final note.

Shapeshifter: One moment you’re head over heels for an artist. The next, you’ve moved on. Some say it’s erratic. We call it eclectic.

Vampire: When it comes to your listening, you like to embrace a little… darkness. You listen to emotional, atmospheric music more than most.

Time Traveler: Have we met before? You travel back in time and listen to songs on repeat, again and again. The best tracks never get old.

Like most of the Spotify Wrapped insights, these reveal the different types of clustering that Spotify performs with machine learning algorithms on our personal music listening habits:

Playlist-based listening habits, characterized by the Alchemist, Roboticist ¹, and Collector ².
Album-based listening habits, characterized by the Hypnotist.
Artist-based listening habits, characterized by the Shapeshifter and Fanatic ³.
Vibes-based listening habits, characterized by the Vampire and Luminary ⁴.
Genre-based listening habits, characterized by the Cyclops ⁵ and Mastermind ⁶.
Song-based listening habits, characterized by the Time Traveler and Hunter ⁷.

By identifying these different streaming habits and quantifying the size of each cluster, Spotify can guide investments into different types of experiences—more algorithmically curated playlists for the Roboticist group, better playlisting tools for the Alchemist and Collector groups, and more.

Listening to less Spotify

As I mentioned earlier, I’ve been trying for some years to listen to less music on Spotify and instead use other services. This year, Spotify made the decision only pay artists for their streams after a track has been streamed at least 1000 times.

From a business perspective, I can see why Spotify wants to do this. It simplifies their payout structures and billing monitoring infrastructure, removing a significant chunk of the long tail of tracks on the service that they’d otherwise need to monitor and pay out fractions of a penny for. All of that calculation and verification takes engineering effort to build and maintain—and for volumes that low, most artists can’t even collect the payouts anyway:

“Often, these micro-payments aren’t even reaching human beings; aggregators frequently require a minimum level of [paid-out streaming royalties] before they allow indie artists to withdraw the money.

It’s easy for me to consider that this doesn’t affect any artists that I listen to, but I went digging through some of my discoveries of this year, and it turned out that one song on an EP that I discovered this year didn’t yet have enough streams to qualify as of November 26, 2023:

As of December 29, 2023, it just barely qualifies:

For artists like Sopp, who I discovered because I.Jordan played him in their Boiler Room set (and Sopp commented on a request for his own track ID), they won’t be able to make any money off that track until it reaches that point.

Metadata repercussions of listening to less Spotify

Spotify has an integration with Last.fm that automatically scrobbles any track I listen to, making the metadata of those tracks extremely high quality. By contrast, songs that I listen to on SoundCloud or YouTube are scrobbled by a web scrobbler extension on my browser, which attempts to parse the description fields into an accurate artist and track, but doesn’t always succeed.

This means that the more I try to move away from listening to Spotify, the more the quality of my personal music listening data declines.

As one example, of the 8,196 songs that I listened to this year (so far), 5,282 of them were unique. Many of those unique tracks were actually YouTube videos that I watched, instead of songs I listened to.

As another example, of the 1435 artists I discovered this year, many of them were artists I discovered previously, but had fancy, inconsistent, or incorrect metadata when they were scrobbled:

02 Daphni
A2 Eris Drew
Anish Kumar & Barry Can’t Swim
Anish Kumar and Barry Can’t Swim
ArcticMonkeys
BONUS Frank Ocean
Bicep “Just”
Catching Flies - GLY (DJ BORING’s Sunrise Mix
Catching Flies, Hot Chip - GLY (Hot Chip Remix
DJ HEARTSTRING | HÖR
DJ Seinfeld | HÖR
Daniel Doherty🏌️‍♂️
Floorplan ‘Never Grow Old’ (Re
Four Tet - Anna Painting (Remix)
Frazer Ray: End Of Year Special
heyrosetta
Skin On Skin // Stay On Sight Recordings
The Blaze play the Cave of the Salamander
Two Shell @TheLotRadio 12
•❀ 𝐃𝐉 𝐁𝐎𝐑𝐈𝐍𝐆 ❀•
“Dang”
𝔻𝔼𝕄𝕌𝕁𝔸

Other artists were also YouTube videos that I watched:

A Day Of Ascents Creates Opportunities For The Puncheurs | Tour De France 2023 Highlights
A Fast And Ferocious Finish Into Bordeaux | Tour De France 2023 Highlights
A Hole Where You’ll Freeze to Death
Accident Case Study
An Intense Day Of Racing On Brutal Terrain! | Tour De France 2023 Highlights
Big Alpine Day With Hair-Raising Descent To Finish! | Tour De France 2023 Highlights
Britain’s Got Talent 2023 Viggo Venn Wacky Semi
Cessna EMERGENCY LANDING ON ROAD |
Combustible Dust
Crashes & Plenty Of Drama On Stage 5 Of The Giro d’Italia Donne
Fake Pilot Travels World
Toby Segar’s Ultimate Ninja Warrior UK Compilation
Uncovered Hazards
world championships

Some of those sound like they could be pretty great artists, but as of today, they’re not.

In an attempt to clean this up, I deleted records associated with the YouTube channels I most frequently watch, such as Mentour Pilot, Taskmaster, or Lanterne Rouge and GCN Racing. That didn’t catch everything, however.

To try to catch any others, I ended up sorting the tracks I listened to by length and cleaning up any tracks that were clearly not song titles ⁸:

That’s not a foolproof way to identify YouTube videos, however, because real songs also have long titles:

At a certain point I just have to go through the tracks manually and remove the erroneous scrobbles. I do this inconsistently, because the volume and utility is such that it isn’t worth setting up a full-fledged data cleaning pipeline, so instead we study it here.

If I ever get to the point where I’m matching my dataset with the MusicBrainz database, that would be an opportunity to build a full-fledged data cleaning pipeline based off the close-to-golden data in MusicBrainz, but I don’t have the bandwidth to run a copy of the database, join it with my Splunk instance, and build a data quality check at index time (or pursue an alternate architecture) as a side project.

And given the flexibility of data uploads and sharing in modern digital streaming platforms, there might not be a golden enough dataset that could accurately validate the available music available across the web.

Calculating a change in beats per minute

One notable difference from my music listening last year is that the average BPM of songs I listen to has changed somewhat dramatically:

In 2022, 121 was the mean BPM, 123 was the median BPM, and 125 was the mode BPM.

This year, in 2023, 131 was the mean BPM, 128 was the median BPM, and a whopping 154 was the mode BPM.

That might not seem like a huge shift, but it hasn’t changed much over the years:

During 2021 the mean was 123, the median was 124, and the mode was 125.
In 2020, the mean was 124, median was 124, and mode was 123.

These BPM values are all firmly in House genre territory, and have been relatively consistent.

I figured these new numbers reflected the fact that I listened to many more high-tempo artists like Mall Grab, KETTAMA, and DJ Heartstring this year, compared with previous years.

When I dug deeper into how I calculate these averages, I discovered that I also have a data problem. I calculate track BPM by regularly building a file with the BPM data that I have from my iTunes library ⁹, which currently contains a scant 143 entries.

With those 143 songs, I do a search to match the songs that I listened to that year with the tracks for which I know the BPM, and calculate the averages accordingly ¹⁰. Using 143 songs to calculate the average BPM for over 8,100 track listens is a bit problematic.

That’s especially clear because, for 2023, it turns out that my average BPM stats are calculated based on ten listens to four songs.

This is another metadata pitfall of using the data easily and locally available to me (Last.fm and iTunes) instead of the vaster troves of data available were I to build a more complex architecture and poll Spotify’s metadata for my most-listened-to tracks of 2023.

But I don’t have to build something super complex to dig into what Spotify’s detailed metadata can tell me. To prove you don’t have to be good at this to improve your data analysis skills, I grabbed a non-random but representative sample of my music listening habits of 2023, using the following criteria:

Top 25 most-listened-to tracks.
Top 25 most consistently listened-to tracks.
Top 25 most-listened-to-tracks by artists I discovered in 2023.

I did some data cleaning because a few of my most-listened-to new tracks ended up being DJ sets, so for LB aka LABAT I swapped in his song Speed of Love, and for DJ Heartstring I swapped in It Ain’t Over and Can’t Stop the Night.

After collecting and de-duplicating those tracks, I ended up with 63 different tracks to collect data for. Me being me, I then opened up Spotify and searched for each track to retrieve the Spotify track ID. In doing so, I discovered that 3 of the 63 tracks were unavailable on Spotify. 1 of those was easily substituted with a similar track, so I gathered data for 61 total tracks.

I gathered BPM data from Spotify using the Get Track’s Audio Features API endpoint, so now I have a dataset of 61 songs that were some of my favorites of 2023 including BPM data.

If I then combine the data from these 61 songs with my listening activity during 2023, I can calculate some new averages ¹¹ and get the following:

A mean of 118
A median of 124
A mode of 131

That’s a pretty wide spread, but it makes sense given that I have some pop songs with some

The audio features are computed using algorithms ¹², and some of the data that I got was suspect in my opinion:

track_name	artist	spotify_id	tempo
I Want To Be Your Only Pet	Bombay Bicycle Club	0bRtjZg3D4IrSJPLj3Jyhu	188.083
My Big Day	Bombay Bicycle Club	0ro5F0G6PaN4hmF3uIibCO	93.263
It Ain’t Over	DJ Heartstring	2CXgBOHvaylFWhzk8aoNPG	145.993
Die Young (ft 347aidan)	Sleepy Hallow	6EbVuwU7EnKeF2hdpViXCH	77.471
Floating Garden	The Nunnery	0poWGgqNWbiaqonEku9De1	74.268
16 Steps	Martin Jensen	7v3MkCal9v1Bw0N31X33qA	180.055

First off, the Bombay Bicycle Club tracks. Both of these tracks, I Want To Be Your Only Pet and My Big Day, are on the same album. I Want To Be Your Only Pet is track two, and My Big Day is track four.

If Spotify’s computed tempo data ¹³ is accurate, that would mean the second track on the album is 188 BPM — 43 BPM faster than DJ Heartstring’s track It Ain’t Over, and more than twice the BPM of a song just one track after it in the same album.

Take a few minutes to listen to all three of those tracks and grasp just how different in speed they are.

The next couple tracks also showcase a discrepancy in the data. Sleepy Hallow’s track Die Young (ft. 347aidan) isn’t a fast track, but 77 BPM is pretty slow.

Put that track on, and then go listen to Floating Garden by The Nunnery. According to Spotify’s data, these tracks are only three BPM apart from each other.

As one last example from this small sample of 61 songs, the track 16 Steps by Martin Jensen is listed as having a tempo of 180 BPM. It’s not nearly as slow as Floating Garden by The Nunnery, but it certainly isn’t 35 BPM faster than DJ Heartstring’s track.

A few years ago, I wrote a long post about why the quality of metadatasets like this one matters for music. The fact that I can find some seemingly incorrect data in a tiny sample of about 61 songs makes me even less confident in the quality of the algorithms attempting to compute audio features like tempo and beats per minute. Tempo quality matters, too, because it can be used for things like creating exercise playlists and recommending more upbeat songs ¹⁴.

Because I did all that research, I also know that there is another, more detailed Spotify API endpoint, Get Track’s Audio Analysis that provides details about every segment of a song. I wanted to see if there were tempo calculations for each segment of a song that might explain these discrepancies, but I found something else — a confidence score for the tempo computation.

If I look at the tempo confidence score for each of the tracks, I get some more valuable insights:

track_name	tempo	tempo confidence
I Want To Be Your Only Pet	188.083	0.144
My Big Day	93.263	0.767
It Ain’t Over	145.993	0.953
Die Young (ft 347aidan)	77.471	0.131
Floating Garden	74.268	0.206
16 Steps	180.055	0.207

In the cases where the computed tempo is much different from my expectations, the confidence score ¹⁵ is also fairly low. So it’s a small reassurance, but at least Spotify knows that their data is bad in these cases.

Digging even further into this, there’s a definite pattern for these outlier tempos. They are substantially faster or slower than expected. As I mentioned, the Bombay Bicycle Club track is just over twice as fast as the other song on the same album. And as it turns out, Sleepy Hallow’s track Die Young (ft 347aidan) is one for which I have BPM data in iTunes—and it’s listed as having a BPM of 154. 154 is twice of 77.

It’s pretty clear to me that while the computations exist to process music and identify the BPM for a track, identifying the dominant beat and thereby the actual BPM (instead of being off by a magnitude of two), is still an error-prone process that could use more humans in the loop for evaluation and validation.

Favorites of the year

Separately from the most frequently listened to artists, songs, and albums, I also want to characterize my favorite artists, songs, and albums of the year. They can easily be the same, for obvious reasons, but there are other factors that influence frequency of listening that don’t necessarily make something a favorite, such as comfort, focus, or other purposeful listening.

Favorite albums

My favorite albums of 2023 were the following:

Overmono with Good Lies. Standout tracks: Feelings Plain, Good Lies, Is U, and So U Kno.
Barry Can’t Swim with When Will We Land?. Standout tracks: Deadbeat Gospel, How it Feels, and Sunsleeper.
Bombay Bicycle Club with My Big Day. Standout tracks: Diving and Tekken 2.
CRi with Miracles. Standout tracks: Losing My Mind, Miroir Miroir, I Can Make It, and Something About.
Pangaea with Changing Channels. Standout tracks: Installation, The Slip, and If. ¹⁶
Sofia Kourtesis with Madres. Standout tracks: Madres.

Favorite EPs

My favorite EPs of 2023 were the following:

DJ Heartstring & Narciss with Why Can’t We Live Forever?. Favorite tracks: Heartbreak Repair System and While U Sleep.
KETTAMA with Fallen Angel EP. Favorite tracks: Feeling Emotions and Fly Away XTC.
Mall Grab with Mouse. Favorite tracks: Loving You, What Do I Do, and Say Nothing.
Y U QT with BABE?. Favorite track: THE WAY U, LET IT GO.
Surusinghe with Brake Fluid. Favorite track: Bop.
DJ Heartstring with SENSUAL HITS VOLUME 1. Favorite tracks: Vision of Ecstasy and Dancing with Somebody.

Favorites that came out before 2023:

DJ Heartstring with 4 The People - EP, from 2022. Favorite tracks: Can’t Stop the Night and Will You Remember Me When I’m Gone?.

Favorite EPs that might be considered singles, actually:

Gemi with Dreams/Tokyo.
Mall Grab with On My Ones/1ofthozedaze.
Aloka with K1/K2, from 2016.

Favorite songs

My favorite songs of 2023, either singles or b-sides:

salute - wait for it
bad tuner - don’t say
Floating Points - Birth4000
KETTAMA - Samba Soccer 2001
Discotron, Disco Junkie, and Sandy’s Groove - Baby Come Back
DJ Cuddles - But You
NOTION - Temperature Dub
NOTION - MS-DYNA-MI-TEE DUB
Elkka - Babe
Sammy Virji - If U Need It
Sammy Virji & Flowdan - Shella Verse (hearing this live in IPC’s set was incredible)
Carly Simon - Why (I. Jordan remix)
jamesjamesjames - back2u
DJ Heartstring - It Ain’t Over
Mall Grab & Flansie - You
Ross From Friends - The One

I also had some favorite tracks that I enjoyed this year, but which didn’t come out this year:

Baltra - Luv-N-Me (from 2022)
t e s t p r e s s - Just (from a DJ set, maybe IPC?)(from 2022)
Prozak - Pump It (from 2022)
Elkka - I Just Want To Love You (from 2022)
Surusinghe - Good Girls (from 2022)
Michael Gray - The Weekend (from 2005)
Sammy Virji - Foundation (with Oppidan) (from 2022)

My favorite pop music tracks were also not from this year, and are also not singles, but here they are:

Dagny - Love You Like That (from 2017)
Griff - Walk (from 2021)
Abby Sage - Backwards Direction (from 2022)
Caroline Polachek - So Hot You’re Hurting My Feelings (from 2019)
ELIO - Waste of Emotion (from 2020)

Some lesser-favorite, but still very enjoyable, tracks from this year, are:

KETTAMA - Rok da House
Sopp - Lift Me Up (from I.Jordan’s Boiler Room)
Skin on Skin - Magic
O’Flynn - Swiss Sensation (not available for streaming)
Roobinz - U Got This
Flansie - closer
Overmono - Blow Out (took some time to get used to)
Oppidan - Remember

Favorite DJ sets

In ranked order, my favorite DJ sets that I did not attend in 2023 were the following:

Overmono Boiler Room in Manchester
DJ Heartstring Boiler Room at AVA Festival
Romy Boiler Room in London
Steel City Dance Discs hosting LOODS on Rinse FM (from 2022)
Skream B2B Interplanetary Criminal @ Beams Recording (from 2022)
MixMag Impact: Interplanetary Criminal (from 2022)
MixMag The Cover Mix: HAAi (from 2022)
LB aka LABAT Boiler Room at FLY Open Air
Mall Grab B2B Skin On Skin @ Junction 2 Festival 2023

The sets by Skream, Interplanetary Criminal, and HAAi are late additions, but they were on primary rotation during my cross-country flights for the holidays.

Wrapping up 2023 in music

It’s been a year of new discoveries, familiar artists, pocket investigations into what makes a genre and how BPM is defined, as well as the mundane streaming habits that Spotify uses to cluster and categorize its listeners.

I hope you enjoy some of my favorites. I’ll be back in another year to dig through my dirty data.

“You like to hit play, kick back, and let the clever algorithms work their magic, track after track. Oh look, that rhymes.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“Your taste is sublime. You listen mostly to your own playlists, and we totally get why. They’re perfect, after all.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“Once you pick a favorite, you never let go. Your top artist makes up more than a third of your listening. Impressive.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“There’s a spark in you, and your listening shows it. You play light, upbeat music more than others. Bet you’re fun at parties.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“When it comes to your listening, you’re loyal and devoted. You like to focus on one genre. Sometimes while wearing a monocle.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“Knowledge is power, listener. Which makes you powerful indeed, as you like to study a wide range of different genres. Clever you.”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎
“You’re always searching for new favorites. You skip tracks more than other listeners. Maybe it’s the thrill of the chase?”, from Me in 2023 Reveals the Streaming Habit That Defined Your Listening. ↩︎

Using the following search:

`lastfm` 
| eval length=len(track_name)
| fields track_name length
| sort -length
| dedup track_name
| table track_name,length

↩︎

Using the following search:

index=music sourcetype=itunes_xml
| search BPM!=""
| inputlookup trackbpm.csv append=t
| table BPM,track_name
| outputlookup trackbpm.csv

↩︎

Using the following search:

`lastfm`
| fields track_name,artist,_time
| lookup trackbpm.csv track_name OUTPUT BPM
| search BPM!=""  
| fields BPM, track_name, _time
| stats mode(BPM) as avg_bpm
| eval avg_bpm=round(avg_bpm)
| fields avg_bpm

↩︎

Using the following search:

`lastfm` | fields track_name,artist,_time
| lookup 2023bpm.csv track_name OUTPUT tempo
| search tempo!=""
| fields tempo, track_name, _time
| stats mean(tempo) as avg_bpm
| eval avg_bpm=round(avg_bpm)
| fields avg_bpm

↩︎

I dug back into the patents to try to determine what mechanism Spotify uses specifically to compute tempo. So far, I discovered that “Techniques for deriving acoustic metadata from music signals are known.”, from Automatically acquiring acoustic information about music. The patent on searching media content based upon tempo references another one, Cadence-based playlists management system, which provides further information:

“A tempo of a media content item can be determined in various known manners. In the example of songs, a tempo of a song can be relatively easily identified because songs typically have a steady tempo throughout their entire playing time. Where a tempo changes significantly throughout a song, in some embodiments, such variations in tempo can be averaged to represent a single tempo of the song. In other examples, a portion of the song having an approximately constant tempo can be identified, and such a constant tempo can be used as a tempo for the entire song. In yet other examples, the portion of the song having an approximately constant tempo is taken and used to replace the entire song while the other portion of the song, which has variable tempo, are excluded from playback. Other methods of obtaining a tempo of a song are also possible.”

Which is a rather dry and depressing way to describe how a song is sampled to determine a tempo value for the entire song, but doesn’t describe how the beats are computationally identified by an algorithm.

Apple has another patent, Musical analysis platform that goes into more detail about identifying key signatures than about identifying tempos, but makes it clearer how one might calculate and analyze beat placement in a song to identify tempo:

↩︎
I don’t know how they compute it, but it’s defined as “The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.”, from Get Track’s Audio Features. ↩︎
Tempo quality specifically matters because it can be used to, for example, craft a playlist of songs for running (as described in the patent for Search media content based upon tempo):

If the computed tempo has a low confidence score, it’s unlikely that it will be used to add the track to a playlist, but the patent doesn’t mention logic that incorporates the confidence score associated with the tempo.

As another example, in a context where the media content might steer playback behavior, such as that described in the patent Media content steering, the tempo of the track also plays a role:

“In other embodiments, the second media content item is selected such that at least one of the other facets remain the same or similar between the first media content item and the second media content item. In this example, the second media content item is determined to have a higher tempo that the tempo value of the first media content item, in order to suit the user request for “more upbeat” songs.”

↩︎
Defined as “The confidence, from 0.0 to 1.0, of the reliability of the tempo.”, from Get Track’s Audio Analysis. ↩︎
I saw Hessle Audio do a B2B2B last year thanks to the encouragement of some “scene elders” and I gotta say, while the set was a bit uneven, it was worth it to make it out there. If you go to shows with folks that are newer to the genre, take the time to educate them on the essentials. ↩︎

Recommended blogs

Sat, 30 Dec 2023 12:03:59 +0000

Inspired by Matt Webb’s post 10 blogs for your newsreader, I decided to share my own favorite blogs to follow.

I also use RSS feeds to consume most of the content I encounter, and always get sad when a site doesn’t provide an RSS feed to let me follow new posts. I end up following many newsletters through RSS as well, because I’m better about keeping up with my feeds than I am my email inbox—or at least Feedly, my feed reader of choice since the demise of Google Reader—removes unread posts that are older than 30 days by default.

These are the blogs where I read, or at least intend to read, almost every post:

Elezea by Rian van der Merwe. Rian does a great job sharing his thoughts about product management and also sharing valuable and fascinating articles having to do with product, organizations, and business.
Simon Willison’s Weblog by Simon Willison. Simon has been thoughtfully and technically covering AI for the last year, also sharing valuable articles that he comes across elsewhere. When he’s not writing about AI, he’s writing about his data tool, Datasette, and other data and content related thoughts.
Things That Caught My Attention by Dan Hon. Writing about technology from a personal, social, and organizational perspective, if you ever think “I should read thinking in systems” Dan is someone who is thinking and writing about those systems.
The Beautiful Mess by John Cutler. Another great product management blog, John writes short and thoughtful product management and product organization perspectives regularly.
Ask a Manager by Alison Green. Most of what I’ve learned about what to expect from my workplace, how to navigate professional norms, and how to look for a job, I’ve gleaned from Alison’s excellent and entertaining advice column.
Labnotes by Assaf Arkin. Assaf has been writing this for years, aside from a brief hiatus when he was hit by a car, and I always enjoy the links and posts that he shares about technology and the world. I follow many newsletters like this and there’s always new and interesting content in what Assaf finds.
Interconnected by Matt Webb. Explorations and wide-ranging perspectives on “the future of technology, design, and society”.
Internal Exile by Rob Horning. Rob used to be an editor for The New Inquiry and Real Life (rip) and now writes this newsletter. I’ve been following his writing all the while.
Monkeynoodle by Jack Coates. I worked with Jack for a little while many years ago, and now I follow his great blog about product management and organizations.

Other favorites that I don’t read as consistently, but I wish I did:

Rest of World which I started following to gain a perspective about tech in countries like China and India. The focus of the site has since broadened, supported by a vision “to become an indispensable source of information that captures people’s experiences with technology outside the West”. Highly recommend, especially their reporting on the folks performing data labeling for AI tools.
404 Media founded this year by some folks from Motherboard and are doing incredible journalism already, especially on data security, privacy, and AI.
She’s a Beast by Casey Johnston. Casey has been writing “Ask a Swole Woman” for years, and wrote an ebook, Liftoff: Couch to Barbell. She’s been on a mission to empower women and others to get stronger, and along the way, trust their bodies and themselves.
★❤✰ Vicki Boykis ★❤✰ by Vicki Boykis. I started following Vicki when she was writing Normcore Tech about ordinary data and the bread and butter of data work. As life happens, she’s largely stopped writing the newsletter but still occasionally posts on her blog.
Counting Stuff by Randy Au. Randy covers the bread and butter of data work as well (He worked with Vicki to put on Normconf with many other talented folks) and it’s so straightforwardly useful.
Untangled by Charley Johnson. Charley writes about the intersections of society and technology and the role of power along the way.
The AI Ethics Brief by Abhishek Gupta of the Montreal AI Ethics Institute. A brief yet thorough overview of research and policy updates in AI ethics and other movement in the space.

Displaying content as a graph: An exploration

Thu, 28 Dec 2023 20:20:27 +0000

Most web content is designed to display with a strict hierarchy, tree-based or otherwise. What if it wasn’t?

What does it mean to display content as a graph?

Instead of being displayed as a strict hierarchy, content could be placed in multiple categories within a hierarchy, using a polyhierarchy, or displayed as a series of interlinked nodes in a network—a graph.

Displaying content as a graph could visually communicate the relationships between ideas, tasks, and topics in technical content. A hierarchy also does this, but doesn’t always match the actual relationships between ideas and concepts in technical content.

A graph could offer a more flexible presentation, but content is usually displayed hierarchically on the web. Why is that?

Why hierarchies are common

Hierarchies are common because they match the typical way of explaining a new concept or topic — you start out with an overview, briefly introducing several important things and how they fit together, and then provide additional information about each thing later on.

For example, if I want to teach you how music is made, I’m not going to start with an intense explanation of how an instrument works to create sound. Instead, I’ll start with a high level discussion about the components involved in music: a song, an instrument, scales, key signatures, tempo, and then move into more complex topics like how to write music using those components.

How to make music
- What is a song
  - What is a hook
  - What is a verse
  - What is a chorus
  - What is a bridge
- What is an instrument
  - Different types of instruments
    - String instruments
    - Woodwind instruments
    - Brass instruments
    - Percussion instruments
  - How to tune an instrument
- What are the components of music
  - Types of scales
  - Melody and harmony
  - Tempo and time signatures
  - Key signatures
- How to write music
  - How to write a hook
  - How to write a verse
  - How to write a chorus
  - How to write a bridge
- How an instrument works
  - What is sound?

It’s fairly easy to scan a list of items and follow the hierarchy that represents a logical progression of information and knowledge.

Hierarchies also mimic a traditional way of displaying information—as a book, with chapters that represent hierarchies of content.

Why use something different?

Hierarchies are common and familiar. So why use something different?

Serve multiple mental models

Presenting content in a hierarchy provides an implicit mental model to the reader. By ordering content, there is an implication that the information is listed according to its relative importance and in the order which you should learn the information.

In the case of the music example, because of the hierarchy presentation, that means that you should learn the component parts of a song before learning about instruments, and before learning about scales, key signatures, or tempo.

If you display content as a graph, you can remove some ordering implied by a hierarchy:

By expanding a hierarchy, either by placing content in more than one location, or by removing the ordering involved in a hierarchy, you can serve multiple mental models used by your readers and thereby improve findability.

An article from Page Laubheimer for Nielsen Norman Group¹, points out some benefits of categorizing content in more than one location:

A polyhierarchical IA is a structure where an item exists in more than one place — that is, it can be reached following several category paths. … Ideally, you want to create categories that accurately describe your content and products, are extensible when you add new topics or products, and match your users’ mental models (which often vary from one person to another).

And that is a crucial point — we often write documentation with an audience in mind, but there is always variation in familiarity and background.

For example, because I learned data analysis with a language that performs aggregation with the statistical transformation command ², I might look for the SQL function GROUP BY alongside the COUNT() and SUM() function reference.

If content could be displayed as a graph, depicting different relationships across categories and commands, the content can reflect many mental models. The content could become the diagram.

Write better documentation

A persistent challenge for technical writers is to focus on writing user-centric documentation rather than feature-focused documentation.

When you consider content with the complexity of a user journey, with all the requisite branches and forks, the limitations of linear hierarchies begin to show.

It could be easier to write documentation if you didn’t have to consider where the content belonged in the navigation or the table of contents, but instead considered how it related to other pieces of content that already existed. We do that already, but must always create a hierarchy that considers order.

But sometimes order doesn’t matter. Sometimes order is different depending on the audience. Hierarchies are inflexible.

Without being constrained by a hierarchy, we could place documentation in a structure more mindfully, answering questions like:

What relationships do you want to draw for the reader?
When and where might the reader encounter the tasks described in this topic?

David Ryan at Red Hat has tried this, pointing out in a tweet that:

A few of us used a graphing tool at Red Hat to show content relationships, as we experimented with moving away from “linear book” and tried to match content with the user journeys.

For the music example, we can acknowledge the fact that some readers might be familiar with writing lyrics, but not familiar with the concept of key signatures or scales. Other readers might be familiar with songs, but can’t keep tempo easily, so want to explore that content in more depth than others.

Improve content reuse

If you don’t have to draw strict hierarchies, it could be easier to reuse content. If your content is pocket-sized and specific to a concept or task, recontextualizing it would be as simple as adding another link to the graph.

In a hierarchy, to reuse content you often have to duplicate the content to another location—through single sourcing or otherwise—to make it logically relevant.

The flexibility of a graph means that you can use labels or links to draw connections across content, rather than duplicating it to another location in a hierarchy.

Improve machine legibility

Content that is not just stored as a graph but also written for a graph would be even more legible to machines—the website crawlers indexing content for use in search result ranking algorithms and capturing datasets to train machine learning models.

Most of that content is already crawled as a graph, with the robots following link after link, indexing the page contents as well as the links between them, building a graphical representation of the web.

Writing with those relationships in mind could help amplify the connections across content, making those connections more legible to the machines while exposing them to the human readers as well.

Depending on how you invest in the quality of your knowledge graph, the search benefits could create a competitive moat for your documentation site—provided the content exists to support the search terms.

Why don’t we display content as a graph?

There are a number of benefits to displaying content as a graph, but very few websites do so. Why is that?

It’s difficult to design

Websites are based on pages — and it’s difficult to expose the underlying structure to readers in a coherent way.

Having a navigation menu at the top or the side of a page is an extremely familiar pattern, as well as “breadcrumbs” to help people maintain an idea of where they were before.

There are a few examples of content graphs on the web already, and they follow some different patterns:

Andy Matuschak has created a form of a graph with his Evergreen Notes ³, which open links next to the content that you’re interacting with, and minimize along the way. In the case of Evergreen Notes:

there’s no index or navigational aids: you’ll need to follow a link to some starting point.

It’s a similar design to what Obsidian implemented ⁴. Obsidian also uses a graph and node-based architecture to relate local text files on your computer with one another. It takes the graph visualization to the next level, but I feel that graphs of this scale aren’t super discoverable, despite being explorable, because they’re so overwhelming. Obsidian also recently introduced tab stacks, another innovative way of visualizing content all at once.

Another site is the C2 wiki, which was described to me as “a discussion site that happened to be shaped like a reference guide.” It essentially creates a map for you as you navigate through it, with the links you open floating next to the origin.

I love this design as a way of keeping track of a specific journey through the content, enabling the reader to build their own workflow out of the content, but it raises a challenge when it comes to sharing links.

In the screenshot above, my pathway ended on the DepthFirst page, but if you open that link, you see only the page—none of the journey that led me there, and none of the nodes in the graph that you might want to explore alongside it are visible—only a few links in the page itself.

These two sites expose the main design challenges:

What do you display on the landing page, or entry point, into a site like this? Most readers encounter documentation from search, but many still browse to the site itself—especially prospective customers or employees looking at the company website.
How do you make the graph visible when viewing a discrete page (or node) in the graph? How do you make the context of the current page visible to the reader?

The landing page option might be to do something similar to many hierarchical documentation sites—display a search bar and some key “entry point” nodes with select branch or dependent nodes displayed.

Making the graph visible could be an interesting design challenge, depending on what you choose to do:

Ignore the graph and show only the node, leaving the links as the only way to navigate to other nodes (and follow existing web browsing patterns without adding new ones).
Preserve a specific journey when sharing a link to a specific page, and offer a “historical” graph view, similar to what Andy’s Notes does.
Provide a zoomed-out view option, where you could minimize a page and view the nearby graph nodes:
Something else entirely!

It’s confusing for readers

Hierarchical content is a dominant web format, and as a result, it’s familiar. Familiar design and layout patterns are easier to navigate because you don’t have to learn anything about how to use the site, you can just navigate using the patterns you’ve already learned — scan the navigation bar, scan the table of contents, and skim the headers on the page or use ctrl+f to search for a keyword in the text.

As Mark Baker points out in Every Page is Page One, people are information foragers. The readability of a graph and the sheer amount of content that might exist in one might not lend itself well to the scanning-first habits that we’ve evolved as web readers.

Instead, if you display content as a graph, readers are confronted with an unfamiliar pattern that could require them to learn new buttons and interactions in order to find what they need. That’s a hindrance to readers of technical documentation, because their goal is to answer a specific question, complete a specific task, or learn about something — not learn how your website works.

In addition, displaying content as a graph lacks a clear hierarchy, which causes another big problem: there’s no clear starting point.

As Kelley Gordon points out in her article for Nielsen Norman Group, Visual Hierarchy in UX: Definition:

If you struggle to find focus on a screen, it’s likely that the layout is missing a clear visual hierarchy.

The page’s visual hierarchy controls the delivery of information from the system to the end user — it lets users know where to focus their attention.

Arriving at a webpage without visual hierarchy and only a graph with nodes can be disorienting for readers.

Most readers would then use another form of exploration, such as search, to find what they’re looking for.

Because a graph lacks a clear visual hierarchy and can be overwhelming, the designs of Evergreen Notes or the C2 Wiki make sense because they reveal the graph as a journey as you navigate the page. Other sites that use graphs don’t usually make them visible as you navigate.

One exception is the classic site HyperPhysics Concepts⁵, which uses a visual graph representation with clickable links in order to help you navigate the site:

This graph is a hierarchical and ordered graph, but less hierarchical than a traditional table of contents, and made visible to the reader. Each node of that graph is clickable, opening a new page with content, or sometimes a new graph:

The Hyperphysics site makes the graph visible but not at the expense of the content. The graph can be annotated and serve as another form to communicate the relationships between the concepts described in the content.

Another exception is the site Every Noise at Once. The data sources for it are sadly now defunct, but the site itself displays not a graph, but a collection of genres laid out as though there were node connections visible.

It’s the idea of a graph, but with no true connections. The site itself refers to it as a scatter plot — so there are vague associations, but no true links. If you go deeper into the graph and select a genre, you see a similar scatter plot of the artists in that genre:

Sites with hierarchies also use breadcrumbs as a way to show you where you are in the overall site map. Content that you interact with purely in graph form would instead need to either construct dynamic breadcrumbs based on the path through the content that you took, or take a different approach entirely, like relying on category tags.

Every Noise at Once, with the flexible graph-like exploration format, maintains a breadcrumb that effectively shows the nodes surrounding the one that you’re currently visiting:

Ultimately, there are a variety of options for displaying content visibly in a graph, but most of them either retain a hierarchy to assist readers in scanning and foraging for information, or have no visual hierarchy and force the reader to explore, use search, or give up in the face of overwhelming options.

It’s too flexible

Content graphs aren’t a new concept. Displaying content as a graph might be rarely done, but storing content as a graph is a common practice—usually referred to as a knowledge graph.

As Sarah O’Keefe discusses for Scriptorium, there is a cost to knowledge graphs. One of those is the challenge to writers:

We have to think about the various content or data objects, understand how they relate to each other at the knowledge graph level, and then bring them together into a coherent experience, whether a webpage, a document, or something else entirely.

That’s a lot to consider. Writers using DITA (Darwin Information Typing Architecture) and XML-based structured authoring solutions might already be familiar with these challenges, and the chunking approach to reduce content to its relevant bits which then need to be reconstituted into a coherent experience on a webpage.

I haven’t worked in a DITA environment before, and most of my experience is with tools that don’t use a database to store content, let alone a graph database. Many docs-as-code tools require a hierarchy of sorts to dynamically generate a table of contents (Sphinx is one example) or to provide some sense of order to the flat file structure of a repository.

When it comes to organizing content, it could be freeing to consider all the different ways a topic could be relevant to a reader, but it could also end up exhausting, deciding where to draw the line.

For the music example from earlier, a page, or graph node, about “How to write songs” could be linked to:

How to write a bridge
How to write a chorus
How to write a verse
How to write a melody
How to write a topline
How to write a hook
how to write lyrics
how to write a harmony
how to use a digital audio workstation (DAW)
and more…

However, the flexibility of a graph could add chaos. If this content could be linked to other content, what should it be linked to? What other nodes are relevant? A graph could quickly turn into a sea of possibilities, rather than intentional choices about relevance.

At a certain point, the attempt to avoid missing a relevant link, or to try to make your content as discoverable as possible through a knowledge graph, would dilute the explicit mental models and diagramming potential possible with a graphical presentation of content.

Instead of replicating some mental models, the graph could instead start to represent all possible permutations of a mental model, and stop representing anything of value at all.

But the act of considering all the different locations where your content might be relevant, what other entry points might exist, could result in more thoughtful documentation content. You can consider it as part of an entire system. By considering the content as part of a graph, you can more easily treat the content as part of an expansive surface, rather than a discrete element on a specifically ordered list.

By contrast, because a hierarchy is strict, it forces choice. If this content had to exist in one place, where is the most relevant place?

That one choice can be daunting, especially when choosing what the top of the hierarchy should be.

In a discussion in the Write the Docs Slack community, Wouter Veeken points out that one difficulty with hierarchies is deciding what to put at the top of it, asking if anyone has a “general method for deciding which attribute should be the top level of the tree?”

Therein lies the tradeoff. For some content, choosing a location in a hierarchy is an arbitrary choice. For other types of content, it reifies a relevant order to a task.

For example, if you want to write a song, you can start with a melody, a harmony, or lyrics. To write lyrics you can start with a chorus, or a verse, or a bridge. None of those have an order. But if you want to produce a song, a song must be written first. In that case, order matters.

This is a key advantage of having a graph structure—everything is a set of relationships, there is no designated “top” of the tree to identify. You could even choose to display highly-related topics differently than others, and use the information provided by the graph structure itself to determine what content is most valuable.

It’s difficult to write chunked content

Those are the challenges with structuring the content, but there are also challenges with writing the content.

Carrie Hane points out in a discussion about Knowledge Graphs on Sarah O’Keefe’s LinkedIn that:

adoption of structured, decoupled content is still needed to adopt knowledge graphs. And that is a huge leap.

I agree with that completely. Each piece of content in the C2 wiki or the Hyperphysics site was a discrete component of content — a definition of a concept, for example. That type of chunked content fits well within a knowledge graph.

Technical content written for a hierarchy, or with an implied order, can often end up quite long and detailed — perhaps with one page describing everything you need to know about something. To continue the music example, you might end up with a page like:

How to write a song
- About song components
- Write lyrics
  - Write a verse
  - Write a chorus
  - Write a bridge
- Write a melody
- Write a harmony
- Write a bassline
- Write a percussion track

That page is full of task-based content that could be easily separated into different graph nodes, but it might also include definition content mixed in, like:

What is a hook
What is a topline
What is a key signature
What is tempo

That definitional content would also get broken out into specific chunks. All of a sudden, instead of writing an end-to-end topic all about songwriting, you’re writing tiny chunks of content that might not be as interesting, compelling, or cohesive to write or read.

Sparse content and dense content don’t scale in a graph

Graphs are only useful if there are lots of connections. A graph with only a few nodes doesn’t communicate much as a diagram of a mental model:

On the other hand, too many nodes is impractical and also doesn’t communicate much as a diagram of a mental model:

Visible graphs: The future of displaying content?

Ultimately, it’s unlikely that visible graphs would become a standard practice for displaying technical content. It’s a complex design and rendering problem, and while there’s a chance that the novelty might get someone to click around in a graph for a bit, someone with a specific question in mind wants to be able to scan for information.

A hindrance to scanning is volume of content. Given the amount of content on many documentation sites, and that people are accustomed to navigating in digital devices, most people default to using search as their first choice for navigation.

If everyone is using search to navigate, displaying content as a graph wouldn’t make much sense.

Instead, implementing a knowledge graph as the background structure for a documentation site, with clear guidance about how to connect the nodes of the graph, has the potential to improve search results for readers and allow flexible categorization for writers.

A knowledge graph lets you have the best of both worlds. As Manav Modi from ProductX points out in How is Airbnb optimising Search and Discovery using Knowledge Graphs? 🎯:

The graph can also be hierarchical, with high-level concepts branching down to more specific details, allowing for a streamlined data organization.

An alternative that can preserve the graph conceptually but not visually is the tile-based approach common on many documentation landing pages, with sort and filter options:

One compelling example is the Redis command docs, as shared by @matheusfelipeog in their beautiful docs repository.
Another one is the GitHub Docs which combines a tile-based presentation with a table-of-contents hierarchy, as well as recommended pages based on “getting started” or “popular topics”.
The Digital Ocean docs also use a tile-based approach combined with a table of contents.
Mode uses tags on their resources page, letting you filter by type of content and/or the subject of the resources.

In these models, the graph isn’t made apparent, but the node relationships still exist through tags or content types and can be explored without using a table of contents. This approach lets you gain some benefits of visually displaying content as a graph without the most troubling challenges—an unfamiliar and challenging browsing experience for readers.

No matter how you visually present your content, the act of placing content within an information architecture and structuring it is valuable and reifies a mental model for the readers and the writers.

It’s impractical to make a graph visible to readers, but I still think it would be cool.

Polyhierarchies Improve Findability for Ambiguous IA Categories ↩︎
SPL, or Search Processing Language used by Splunk software. ↩︎
Thanks to Kimberly for sharing this site with me. ↩︎
Thanks David Ryan for sharing this site with me. ↩︎
Thanks Scott Centoni for sharing this site with me. ↩︎

Should you add screenshots to documentation?

Sun, 10 Dec 2023 21:15:21 +0000

Screenshots in documentation can be a contentious topic — some people really like them and think they add a lot of value, while others dislike them due to the maintenance burden and accessibility issues.

As a technical writer, I avoid adding screenshots to documentation as often as possible. In my mind, an outdated screenshot is one of the fastest ways to lose customer trust, so if I’m not confident I can maintain the image, I don’t add it.

But that approach might be too rigid and avoidant. What are the advantages and disadvantages to adding screenshots to documentation?

Why screenshots are bad

Screenshots in documentation can have a lot of problems:

Outdated screenshots can cause customers to lose trust in documentation accuracy.
Pages load more slowly when they have a lot of screenshots compared to pages that only have text.
Screenshots can replace step-by-step instructions, losing an explanation of why you might perform a task and replacing it with an image of how or where.
Screenshots often have no or poor alt text, making them inaccessible to those with low or no vision.
Relying on screenshots can lead you to write documentation that only describes the UI, instead of what you can do with it or what the data visible in a dashboard means and where it comes from.

However, these are mostly implementation problems. It’s possible to have screenshots in documentation without these problems.

Why screenshots are good

Screenshots and images in your documentation can serve some helpful functions:

Make it easier for readers to scan to a relevant part of the content.
Provide a visual frame of reference for people reading the documentation without using the product at the same time.
When used correctly, they can supplement confusing task steps.

Indeed, the Baymard Institute performed user research about SaaS products and found that 35% of SaaS Sites Fail to Make the Service’s UI Sufficiently Prominent to Prospects, making it harder for potential customers to learn more about the product. The findings apply to a product’s entire website presence (beyond the product documentation), but it’s important to keep in mind when deciding whether to add screenshots.

In light of those findings, it’s unfortunate that SaaS products typically update the UI constantly, making it especially hard to maintain accurate screenshots.

But these are some real advantages to providing screenshots in your documentation. So how do you add them while still avoiding the problems?

Identify the purpose of a screenshot

If you focus on the purpose of a screenshot in product documentation, you can identify a few situations where screenshots make sense:

Make a confusing page clearer by adding context to it.
Give someone a sense of how a page might look when it has data in it.
Walk users through a task that traverses multiple different pages in the UI.

In other cases, a screenshot might function as filler and not provide much of a function while still being difficult to maintain. Don’t add screenshots in these cases:

For every step in a task.
To provide visual interest without a reference in the text.

You also don’t want to provide screenshots to support text that describes every element on a specific page, because you don’t want to write documentation that only describes the user interface ¹.

Make screenshots accessible

Whenever you add a screenshot, make it accessible. Accessibility of images is relevant for lots of different people:

A data analyst with low vision that keeps the documentation site zoomed in at 400%.
A sales engineer using a mobile hotspot to work at the airport.
A sysadmin visiting family in a rural area that just got woken up by an urgent page in the middle of the night.
An engineer using a localized version of your product in their own language.

If critical content is only available by looking at an image, it isn’t accessible to any of those people. Therefore, make sure that a screenshot isn’t the only way to gather important information.

You can start improving image accessibility by writing useful alt text. Refer to Microsoft’s excellent guide, Everything you need to know to write effective alt text.

You also need to consider the file size of any image that you add to your documentation, because image size can affect page load. Compress your images using a tool like compressor.io (or others compiled by Smashing Magazine) before adding them to your documentation.

If you want, you can work with a translation service to localize images in your translated content as well, but you can also use other approaches to help make them accessible to folks who are reading your documentation other than the language your content is written in, such as by simplifying your screenshots.

Use simplified screenshots

Being intentional about adding screenshots to your documentation helps reduce the maintenance burden because you have fewer screenshots to maintain. But if you use simplified screenshots, you can reduce the maintenance burden further.

A simplified screenshot is a screenshot modified to obscure or remove content irrelevant to the purpose of the screenshot. Anton Bollen introduces the idea in his excellent article, Simplified graphics: Meet the new design style for technical communication.

For instance, you might provide a screenshot with some content blurred out, to help readers focus only on the relevant portion of the image.

While they might take a bit longer to make, simplified screenshots don’t need to be updated as frequently because you can obscure parts of the UI that are irrelevant for your task, unavailable to users with specific roles or permissions, or that might otherwise be confusing.

With a simplified screenshot, it becomes outdated at the same time as your text content, so you can update them both at the same time—rather than your image becoming outdated sooner because some other part of the interface changed.

Automate screenshots

If you’re committed to providing screenshots and have a robust docs-as-code pipeline, you can also consider automating screenshot creation and maintenance.

Simon Willison wrote and released a tool called shot-scraper which you can use to automate taking screenshots of webpages (and by extension, web applications). I haven’t used it, but I can envision designing a screenshot pipeline that works as follows:

Attach a hook to a UI component that you use in a screenshot.
If the code changes, prompt the tool to automatically take a screenshot in a demo environment.
Compress the resulting image.
Create a PR to add the new image to the documentation, replacing the old image.

Of course, componentized UI code means that this could be pretty difficult to instrument on a large scale (the only scale at which this would be necessary), but depending on your code base and technical skills, this might be feasible for your documentation.

Intentionally add images to your documentation

If you have a strategy and a process for adding and updating images in your documentation, you can avoid the pitfalls of screenshots in your docs and bask in the advantages.

Add screenshots to the documentation only when they’re relevant and useful.
Make screenshots accessible, so even if they aren’t there, the content is still useful.
Use simplified screenshots to ease the maintenance burden and increase the relevance.
Consider automating screenshots if you have the resources.

In rare cases, you might be writing UI reference content, such as when documenting critical workflows for a process where the UI has just had a major update and buttons have moved around, or if you’re documenting software for users that have never used software before. But otherwise, if you find yourself writing content that describes the available buttons on the UI, you might want to consider your audience more fully. ↩︎

Improving documentation findability in an age of low-quality search results

Tue, 05 Dec 2023 20:57:56 +0000

For months, there have been reports on deteriorating search quality ¹. As the quality of search results deteriorates, so too does an important factor that makes technical documentation so useful — its findability.

In an era of web-first authoring and product-led growth marketing strategies, organic documentation findability matters. For some software documentation sites, at least 80% of traffic to the site comes from organic search.

You don’t want to abandon all efforts at optimizing for search engine results ², but it might be time to reduce our reliance on search as the primary front for documentation findability.

If we stop expecting readers to discover documentation on their own using search, how do we as technical writers make sure the information can still be found? What’s next for documentation findability?

A common tactic to improve search result quality is to append “reddit.com” to get higher quality results. The results are higher quality because they’re human-curated and generated, rather than algorithmically surfaced ³.

Reddit is just one site with human-moderated and curated conversations — your community is also having discussions in Discord or Slack communities, in the comment thread of a Substack newsletter, in person at events, and more.

If you support and engage with your community by responding to documentation feedback and requests, your community members show up to these conversations and discussions with awareness of the documentation—not just their most-referenced pages, but recently updated and published ones as well.

Why does this matter? I always say that my “north star metric” of documentation success is “if someone asks a question and a link to the documentation can answer it”.

If you write high quality documentation, and build champions of your product (and your content), those champions bring your content to the product and business problem conversations happening across the community.

That sounds amazing — but how do you make sure those champions find your content?

If you document a product with a user interface, you likely have some amount of in-product help. Oftentimes, that can be pretty basic.

For example:

A tooltip to describe something.
A “learn more” link in an error message that goes to the documentation.
A sentence of explanatory text in a dialog box that links to the documentation.
A link to the documentation.
A search box in the product that lets you search the documentation.
A help command in a command-line interface with varying amounts of information available.

As interfaces get more complex and documentation content gets harder to find on the web, it’s beyond time to think beyond the tooltip about other ways to incorporate documentation into the interface in a helpful, maintainable fashion.

We’ve all encountered the overly aggressive product tour when using a new web app, “helpfully” highlighting different parts of the interface the very first time you open it, either distracting you from your intended task (and getting in the way) or showing you about new features when you’re not even sure what your task might be yet (just trying to explore).

Instead, some other modalities could help:

Interactive tutorials that use the product to explain it — example code or templates.
A “show me how” option that demonstrates a discrete task on demand, rather than a tour that appears on first page load.
Autocomplete for commands with in-line guidance about required arguments and parameters.
More robust CLI-based assistance.
An API parameter that provides usage details for the endpoint.
High quality error messages.

Not all of these modalities are new, but they are still easily neglected!

Having help available in these areas isn’t enough. If your command-line tool has a help interface, does it list the available arguments for a command with no context, or is it descriptive and useful? If your API endpoint returns error messages, how helpful are they in helping the recipient fix the problem?

It’s expensive to create and maintain these content types, but if your documentation team is already integrated with your design and engineering teams and processes (hint), then producing this content extends, rather than transforms the way technical content is produced at your company.

If your documentation team is focused on putting out the latest fire or picking up what was thrown over the wall, you might need to invest more deeply in what content and customer assistance looks like at your company to provide some of these in-product modalities.

The closer your content teams are to the product design and engineering process, the easier it is to enshrine consistent terminology usage and avoid mismatches with mental models.

These modalities aren’t just for “product growth” initiatives—they’re about customer enablement and support, helping customers use and learn more about how to do things with the product they pay for—just like the documentation does.

Embrace AI: Chatbots and more

In the spirit of meeting your customers (and prospective customers) where they are, you need to embrace AI.

By this, I don’t mean “train a chatbot on your documentation content and make it available to customers”, because that adds a new interface that customers need to discover, learn how to use, and which you then need to maintain ⁴.

Instead, focus on making your content easily consumable by a language model as training material. That might look like:

Write succinct and clear content.
Use consistent terminology.
Structure your content consistently.

While large language models don’t need to be trained on structured content, the token-based retrieval methods used to generate text rely on vector similarity and associations. If your content contains consistent patterns then it’s likely that the computed vectors for relevant content in your documentation are going to be more accurate.

If your content is used to perform retrieval-augmented generation when prompting a large language model to generate a response, if the chunks created as part of that process all contain relevant information, the results are more likely to be relevant as well ⁵.

For example, if your content is always structured with this pattern:

Enable updates for your device

To enable updates, you must have admin privileges on your device.

To enable updates, do the following:

Log in as an administrator.

Navigate to the system settings.

Select system > general > updates.

For enable updates, select the checkbox.

A language model prompted to retrieve information about the permissions required to enable updates can more easily do so than if your content is structured like this:

To adjust system settings, you must be an administrator.

The available system settings for your device include:

Network

Audio

Appearance

Privacy and Security

General

Battery and Power

It’s important to keep your device updated at all times. It protects you from security threats like malware or viruses. You can set this up automatically if you turn on the correct setting. You can also adjust other settings that can help with security and privacy.

You might notice that the less-structured example is also hard to read and confusing. I’m intentionally exaggerating this example, but the more clear and concise the content, the more helpful it is to your readers and to LLM tools performing retrieval-augmented generation.

Improve documentation findability by going beyond the page

Ultimately, the best way to make your documentation easier to find is to put it where your customers and prospects are looking for helpful information:

Support your community members sharing links to your documentation.
Share content inside product interfaces (web and otherwise).
Implement consistent language and patterns to help readers and LLMs retrieve relevant information.

Take your content beyond the webpage and stay helpful!

Reports such as: June 2022, The Open Secret of Google Search by Charlie Warzel for The Atlantic. May 2023, What happens when Google Search doesn’t have the answers? by Nilay Patel for The Verge. June 2023, A storefront for robots by Mia Sato for The Verge. August 2023, Google’s Search Box Changed the Meaning of Information by Elan Ullendorff for WIRED. ↩︎
Mostly because the best way to ensure high-quality search results for your documentation is to write high quality documentation that is user-centric and task-oriented, and therefore matches the tasks and keywords that readers are searching to find your content. ↩︎
To be pedantic, algorithms of course are involved in who sees what on Reddit, but the responses and posts in any given subreddit, if well-moderated, are from real people. ↩︎
If you want to create an LLM-based tool that uses your documentation content, consider how to make it findable. For example, you could create a GPT in the OpenAI marketplace that is enhanced with a vector database full of tokens from your documentation site and can answer questions about your product—and perhaps even perform tasks in your product. You could also enhance autocomplete in your product with documentation content that is retrieved using an LLM integration. Think beyond the chatbot. ↩︎
I’m making this assertion based on the blog post RAG Isn’t So Easy: Why LLM Apps are Challenging and How Unstructured Can Help, which is confusing to read because the company publishing it is called Unstructured. In the post, the author demonstrates how chunks created with their tool happen at semantically relevant spots, such as before a heading, thus finding that “The RAG system produces higher fidelity responses with Unstructured chunking because the chunks have more consistent semantic meaning, resulting in more relevant query results.” ↩︎

The format of your online technical content matters

Wed, 29 Nov 2023 22:59:33 +0000

How you choose to make technical content available online sends a message about how you do business and what you think about your content and customers…

🌐 Available fully for free on the web: You want your technical content to be as easy to find as possible. You want it to be easily searched, shared, and copied. You don’t mind if competitors know how your product works, because you also want prospective and current customers to know. You want your product and its technical capabilities to be part of the online and offline conversations. You can easily update the content and make sure that people access the most recent version.

🔐 Available on the web, but only if you log in first: You fear your competitors and think that secrecy is your product’s competitive advantage. Existing customers can access your content, but only them. Prospective customers can read your marketing content and talk to sales. If they aren’t paying you, they can’t have a conversation about what your product really does.

📄 Available only as a PDF: You want to make your content available on the web, but just barely. You want to pretend your content is easy to find, but it isn’t easy to search for the content inside the PDF, share a specific part of the content, copy information from it, or link on the web. If you publish content as a PDF, you don’t want to be part of the conversation about the content—you don’t care how people get or share the content, or what questions they might have about it. It’s a bit onerous to publish a new version of the content, and difficult to ensure that people that accessed the earlier versions of the PDF get access to the latest updated one.

With all this in mind, I personally don’t understand why Meta chose to publish their Responsible Use Guide for Llama 2 as a PDF, instead of as part of the user guide for the model. Their own user documentation is reduced to pointing to the PDF and instructing readers to “refer to pages 14–17” of the PDF.¹

Maybe there were some legal restrictions preventing Meta from publishing the guide as part of their official documentation for Llama 2, but it’s certainly sending a clear message—Meta wants you to have information about responsibly using the model, but they don’t care if you find it, or what you do with it.

Fun fact, I couldn’t link to the header for the guidance about reducing hallucinations because the header is actually a <p> tag and not a true HTML header. If you’re going to make content available on the web, please use semantically correct HTML. ↩︎

Why web design sucks now

Thu, 26 Oct 2023 22:05:43 +0000

Heather Buchel’s post It’s 2023, here is why your web design sucks about the current state of web design (and web app design) resonated with me, especially this quote:

“Design decisions can only be pushed so far to the left before we realize the system is broken”

If you bisect design and development into different professions, you can end up with designers that aren’t technical enough to design what’s possible, or developers without enough design prowess to make design decisions independently.

I wonder too if tooling has something to do with it — in an era of Figma and Canva and Webflow, it’s easier than ever to design websites and web applications without having to interact with HTML, CSS, or JavaScript, let alone the DOM of a web browser and poking around in DevTools to get something to display properly.

A similar issue can happen with technical writers in a “throw it over the wall” documentation culture, where a writer relies on functional specs, design docs, and engineering requirements documents to write content rather than actually using the product.

Part of it could be a hiring prioritization motive, where engineers are the first and most-valued hires, while designers and technical writers are seen as support staff and hiring is calculated according to team:designer or team:writer ratios, rather than a partnership-based approach.

In my opinion, cross-functional partnerships with empowered, collaborative product development, are crucial for building a high quality product that customers can use.

How to add documentation to your product life cycle

Wed, 06 Sep 2023 10:53:52 +0000

As a tech writer, I’ve encountered a number of different processes that teams and companies have used to add documentation to their product development processes. Some of these are intentional and others are incidental—but all are used to create documentation across the software development industry.

At its most basic, the product development life cycle involves the following steps:

When you add documentation, that might look something like this:

The typical models I’ve encountered working as a tech writer can be categorized as follows:

The “throw it over the wall” model

The “throw it over the wall” model happens when engineering and documentation teams don’t have much interaction. Instead, in a waterfall-like approach, the development team creates something and tells the documentation writer about it after it’s built.

This can happen intentionally or incidentally. Maybe an entire feature gets developed, is ready to launch, and someone lets the documentation writer know—before or after the release.

Advantages

The advantages to this model are largely to the organization:

Can work well across time zones where the engineering team is in one location and the writer is in another, because you don’t have to try to do synchronous work like attend the same meetings.
Faster to write documentation because the product or feature is fully functional when the writer starts drafting. Encountering bugs during the drafting process can create a feedback loop where the writer can help improve the product and maintain (or build) empathy for the reader, but it definitely slows down drafting when key functionality is broken or not developed yet.

Disadvantages

The disadvantages of this model are both to the customer experience:

Little to no ability to improve or change the product in response to documentation feedback, because development is already complete. At best, the writer can file bugs or stories to get picked up later if they are validated by customer feedback. // for the product
Inconsistent feature coverage in the documentation because only the items that get thrown over the wall as part of a request for documentation get documented. This can lead to content drift, where the docs no longer match the product in some cases, or are missing some features that could use documentation.
Delays between feature launch and published documentation. Depending on when the feature is thrown over the wall, you can get delays between launching the feature and publishing documentation. If that happens, customers might not find out about the new feature, or if they do, don’t have any documentation to help them if they’re confused or something is broken, forcing them to file a support case if they can. Ultimately, this delay can make customers dissatisfied with your product.

And to the organization and its writers:

Unpredictable workload for the writer. By not being involved in the planning process and finding out about a project when it lands on the other side of the wall, there’s a constant juggling of priorities with little flexibility on timelines. If every project appears as a surprise, it’s pretty difficult to plan resources or estimates.
Pressure to write documentation quickly because documentation was left as the last step in the process, they’re seen as a bottleneck to the release process.
Low context about the product and low rapport with the team. You might produce documentation that misses key functionality that wasn’t apparent to you when you encountered the functionality, or not have a full understanding of what was built or why, depending on what the handoff process looks like.

The “put out the latest fire” model

This is a fairly common model, especially for overburdened documentation writers or those working as contractors or consultants on a project. You could be an external freelancer brought in to work on a specific project or be part of a documentation or content team that works with others on a project-by-project basis.

In this model, you join a project and work with that project team in an embedded or throw-it-over-the-wall fashion until the doc needs are met. You might only have one point of contact, like a product manager, and you deliver the documentation requested by them. After the project concludes, you join a different project.

Advantages

The advantages to this model are largely to the organization:

Lets fewer tech writers work across more projects. Whether it’s successful is debatable, due to the volume of context switching required by the writers.
Faster onboarding onto new projects or teams, since the onus is on the project team to build the context for the writer, often to the degree of writing drafts or sending screenshots to the writer.
Can permit flexibility in commitments, where projects without allocations go intentionally undocumented (or get delayed) because the documentation team doesn’t have the resources to support a requested project, or you can move a writer from one area to another

Disadvantages

The disadvantages are borne by both customers and the organization:

Results in feature-driven documentation rather than user-centric documentation. Because the writer has less context on the what, why, and for whom of the project, you can end up with feature-driven documentation that describes what was built rather than what the reader can accomplish with the new functionality.
Estimating documentation workload is challenging due to the overlapping and intersecting projects, combined with the difficult-to-anticipate effects of context switching. More time can be spent juggling priorities than writing, and writers can quickly end up overburdened compared with their colleagues.
Needs org-wide process consistency to be successful. If project teams don’t work using the same (or at least clearly documented) processes and rituals, the onboarding process onto a new project or team can be jarring and slow, somewhat erasing the benefits of moving writers to other projects as they’re needed.
Docs are unowned and maintenance gets left behind. Because you leave a project as soon as you ship the documentation, there isn’t much of a concept of long-term ownership, just whoever worked on something last. Without ownership, maintenance of the documentation content is ignored in favor of the latest new feature work.

The “I’m with the team” model

In this model, the documentation writer is treated as a full-fledged member of the engineering team, involved in all the same meetings, Slack channels, and most discussions.

Advantages

The advantages of this approach are mostly for the customer:

User-centric documentation rather than feature-driven documentation. If you know what is being built, why, and who for, it’s a lot easier to craft documentation that is guided by the user goals guiding product development rather than writing down what was built and how it works.
Increased consistency in product text (and improved product quality). Because the writer is involved throughout the development and design processes, they can contribute to text in the product—whether UI text (content design) or API specifications and descriptions.
Accelerated content development. With ultimate context and involvement on what is going on, and little if any context switching, the writer can produce documentation fairly quickly and without too much delay before and after development and testing ends.

Disadvantages

The disadvantages of this approach are borne largely by the organization:

It can be resource intensive. It’s often not economical to have a 1:1 writer to team ratio because not every project that engineering works on requires documentation. Of course, if engineers are working on technical debt, writers can also spend time on documentation debt or professional development, but this sort of luxurious level of staffing isn’t one that I’ve ever seen. Depending on the pace of development, the optimal writer ratio might be closer to 1:3 writer to teams, but that’s still higher than the other models.
Writers can waste effort and get confused if they start drafting before the product is finished “enough” to start drafting. If you document every iteration of a feature flow before it is finalized, you might write the same procedure multiple times. Sometimes these drafts let you get to the most efficient and concise way of explaining something, but there’s also a risk that a writer gets attached to an incorrect mental model and the procedure ends up less clear than it might have otherwise.
Can introduce single points of failure. If every writer is embedded in a specific team, there is a risk that if that writer goes on vacation or quits, no one else will have enough context to cover for them effectively.
The writers can be siloed and lack visibility into other projects at the company. Because the writers spend more time with their engineering team(s) than the documentation team, this approach takes extra effort to build documentation team cohesion and consistency across the writing styles on the team.

Which model is the best?

It depends on your priorities and circumstances!

In an ideal world, I think that the “I’m with the team” model is the one most likely to produce the best documentation — but it’s also the most resource-intensive, and therefore might not be the most feasible option for every organization.

The “put out the latest fire” model can work well at a company with low-complexity projects and consistent organization-wide procedures, making it possible to plug-and-play writers on demand to different projects. Unfortunately, many projects that require documentation also benefit from strong domain expertise and local project context, so this approach is best suited for short term transition phases at a company. After you know what projects are going to stick around and need writing commitment, you can invest in a more proactive and embedded approach like the “I’m with the team” model.

For small companies like a seed-stage startup, or one that is great at asynchronous communication, the “throw it over the wall” approach lets you add documentation without changing your core engineering processes. If you’re at the early stage where you’re prioritizing moving as fast as possible most of the time, this approach probably makes sense.

Don't replace your user community with an LLM-based chatbot

Wed, 21 Jun 2023 20:59:46 +0000

There’s been some discussion about the challenges of implementing LLMs, but I haven’t seen any comments about the effect a large language model-based chatbot could have on a company’s user community.

Implementing an LLM-based chatbot seems like an excellent way to do two important things:

Help people find information about your product
Collect data about the information people want about your product

Using a technology-only solution to achieve those goals seems like the ultimate, low-cost way to grow product usage and awareness. But there’s another way—building a strong user community.

Like LLM-based chatbots, user communities aren’t free—they require investments in tools and people to help cultivate the community with in-person user groups, dedicated forums, and/or chat apps like Discord or Slack. Some companies neglect them, leaving the conversations to occur only in the chat apps of related products, or social media.

But even a weak user community offers more benefits than an LLM-based chatbot. An article from Zendesk points out the key benefits of user communities:

Brand loyalty

Evergreen self-service support

Cultivating brand advocates

Social proof [word of mouth]

Customer insights

Brand loyalty

The brand loyalty component is what makes a user community, and your customer base, more resilient (and yes, scalable). You might turn to an LLM-based chatbot to scale customer support or deflect more support cases, but a user community can do that too, with the added bonus of building personal connections along the way.

When your customers interact with each other, and with employees of your company, they build relationships—and those relationships keep people returning to user community hubs like chat communities on Slack or Discord, forums like StackOverflow, or even subreddits on Reddit. Personal relationships motivate people to attend conferences that are focused on your product or service, but are also social occasions.

If your product is good, the personal connections that customers build with each other and with the people building your product can also build loyalty to your product.

Self-service support

It might seem like LLM-based chatbots can offer self-service support—and they can, to a point. It all depends on the training data and the prompted context that the chatbots have access to.

Many LLM-based chatbot implementations rely solely on the product documentation content to answer customer questions—but documentation doesn’t contain instructions for every possible use case, but rather the most common ones (it would be overwhelming to read and maintain otherwise).

For obscure and specific use cases, asking a question in a community forum can often provide a more helpful interaction. For cases where a customer has a mental model mismatch about how the product works, interacting in a chat with actual people who can ask thoughtful follow-up questions can offer a more fruitful interaction than an LLM-based chatbot that can only provide output, and no follow-up questions.

Take this question asked on StackOverflow, ostensibly about ChatGPT token limits:

The top answer takes the time to educate the question author about the difference between ChatGPT and the GPT model that the question author is actually using:

In this case, the language in the question made it clear that the author had an imperfect understanding of the problem space they were working with, which led to a different type of answer.

The response from an LLM can’t reframe and recontextualize the problem in this way. If you’re trying to find information about something but struggling to articulate your question, only a person can help you.

With enough quality training data, such as from a carefully curated and labeled dataset from an internally maintained community forum of questions and answers, an LLM-based chatbot might be able to mimic this clarifying mental model behavior, but it would still be limited.

This strong user community can serve as an informal sales force talking about your product’s capabilities on websites that persist the content, like StackOverflow or Reddit, that can help drive new customers and new business. Meanwhile, an LLM-based chatbot can only help customers that find the tool and ask questions that they know they have. There’s no passive discovery.

Social proof, or the recommendation of your product by trusted experts in the field, is a valuable source of new customers (and reinforcement of existing customers) — one that you can’t buy your way into, or paper your way into with content marketing.

As one example, a question on r/Splunk about whether Splunk is worth the cost yielded this response:

As another example, a question on r/datascience about people’s favorite experiment tracking tools provides a lot of recommendations for someone that might want to start tracking machine learning experiments but isn’t sure where to start.

If you’re idly looking for a tool to use but aren’t even sure of your requirements and stumble across this conversation, you might realize that you also need a tool that integrates with existing projects and look further at Weights & Biases:

As an employee of one of these companies, you can respond to these questions, but it’s less trustworthy than when a real customer responds with their experiences and opinions.

Customer insights

The data you get from a LLM-based chatbot can provide customer insights—rich information about what people want to know about your product, or what people are struggling to find information about, or the problems that they’re trying to solve. If you require users of the chatbot to log in (as Netlify does), then you also gather information about who the people are (although you limit the users to existing, rather than including prospective, customers).

When you build a user community, it’s easier to get quality product feedback from customers that you understand. You can collect feedback about your product or your documentation and build relationships with trusted partners as well as average customers that might be too “small potatoes” to garner special concessions from the product or support teams, but provide a dedicated user base that can expand their usage of your product.

Not only that, but you can get immediate interactive feedback if you interact with your community in a chat app like Discord or Slack. Rather than a data dump that you might need to make time to review and analyze from an LLM-based chatbot, you can engage directly with your customers where they are, and learn more about their use cases in the moment.

While you clearly get customer insights from an LLM-based chatbot, you miss out on most of the benefits of a user community to your business.

Don’t hyperfixate on chatbots

If the community discussions and Q&As about your product get routed largely or entirely to an interaction with a Large Language Model, you have lots of data but no true user community. You might find yourself (and your customers will too) staring at comments stored in a database instead of talking to people.

Without the personal connections crafted in a user community, you miss out on direct customer feedback sourced through genuine conversations—ones where you can ask questions too—and lose an opportunity to build brand loyalty. No one feels loyal to a chatbot.

A chatbot can offer self-service support to customers, but only to those that are already asking questions—and only if the answers to those questions are already written down.

If you implement an LLM-based chatbot at your company, don’t do so at the expense of your user community. Otherwise, the longevity and quality of your product will suffer.

Documenting machine learning models

Thu, 06 Apr 2023 10:46:19 +0000

Products use machine learning and “artificial intelligence” to do things like recommend a song to listen to, offer a quick response to an email, organize search results, provide a transcript of a meeting, and more. Some products rely on ordinary data analysis to construct insights about things like your business performance in a market, or the conversion rates for your online shopping site.

Unfortunately, the companies providing these products often gloss over the technical details of how these systems work—making it seem like those tools are magic, omniscient, or just plain inscrutable. Algorithms are so common, yet it’s just as common for the logic programmed into the systems to be hand waved away.

As integration of machine learning, artificial intelligence, algorithms, and data analysis becomes standard and expected in software products, internal and external documentation for those systems must also become standard.

As a technical writer, it’s perhaps not surprising that I think documentation is important.

Writing documentation is a way to communicate information about your product—and that information in turn lets others use and understand your product.

Commenting code, providing READMEs, writing how-to guides — all of these forms of documentation help people understand and interpret your code, evaluate your project, and use your product.

The normalization of data-driven software, where machine learning models drive key product functionality, means that folks in operations, procurement, legal, and more departments need to understand the components of the product, how it might interact with their system and users, and what risks the business might face as a result.

That makes it extremely important to document each component of a data-driven system, from the datasets involved in data analysis and machine learning model training, to the machine learning models and model results, as well as the systems in which the machine learning models exist in.

When evaluating software, product documentation is expected. It would require an immense amount of trust, and likely some promises, to buy or even start using a free software product that lacks documentation. It’s also much faster and easier to start using software if it’s well-documented.

But machine learning models and datasets available on the web are often not well-documented.

Sometimes that can happen because you encounter a machine learning model without realizing it. You’re listening to music and a model is queuing up the next recommended track in Spotify’s autoplay, or you’re browsing the web and you read an article generated by a large language model. Other times, you’re more aware that you’re interacting with a model, such as when you feed a prompt into Midjourney or Stable Diffusion, or ask ChatGPT a question.

Much like Apple might want you to feel when using their products, artificial intelligence aficionados want you to be able to use their trained models without the friction of documentation about how it works.

But documentation can be vital to make AI-based products more useful. For tools based on generative models, like Midjourney or ChatGPT, understanding what led to the output that you see can help you tweak your input to produce more useful results.

This holds true for data analysis and other machine learning implementations as well, such as any data analysis project in an organization, or internally deployed machine learning models supporting business processes and more. Documentation can provide a lot of support.

Randy Au makes the case for documenting a data analysis process in his post Data Science Practice 101: Always Leave An Analysis Paper Trail:

Despite how much busy work it sounds like right now, you need to leave a paper trail that can clearly be traced all the way to raw data. Inevitably, someone will want you to re-run an analysis done 6 months ago so that they can update a report. Or someone will reach out and ask you if your specific definition of “active user” happened to include people who wore green hats. Unless you have a perfect memory, you won’t remember the details and have to go search for the answer.

Analysis deliverables are often separated from the things used to generate it. Results are sent out in slide decks, a dashboard on a TV screen, or a chart pasted into an email, a single slide in a joint presentation for executives, or just random CSV dumps floating around in a file structure somewhere. The coupling between deliverable and source is non-existent unless we deliberately do something about it.

Without a clear definition of how an analysis was performed, how data points were defined, or where a particular chart came from, the results lose value, credibility, and reproducibility. It helps the future version of you, as well as folks that you collaborate with, if you document the context of a project and store that documentation alongside the output of the project.

Similarly, without a clear sense for how a model was trained or why it might be producing specific results, the results lose value and credibility. In an era where generative machine learning models output fabricated academic references when you ask it for citations about a topic, documentation about the systems becomes even more valuable to help you judge which machine learning model output you can trust.

You can more accurately assess if the outcomes delivered by your model are valid, or due to making the error of testing on your training data, if you have clear documentation about how the model was trained and on which datasets.

As Emily Bender points out in the Radical AI podcast episode, The Limitations of ChatGPT with Emily M. Bender and Casey Fiesler:

If you don’t know what’s in the training data, then you’re not positioned to decide if you can safely deploy the thing.

Especially if you’re using a machine learning model in a commercial context, you want to be able to identify possible harm that could arise from the output, such as violent, misogynistic, racist, or other dangerous output. Without information about the training datasets, model training practices, and overall system context for a model, you can’t properly evaluate it.

Provide helpful documentation

What does it mean to provide helpful documentation for the components of machine learning and data analysis practices?

As with any documentation, you need to consider the following:

Who is the audience?

The audience of the documentation depends on the context of the analysis or the model. The documentation for a dataset distributed on the web for free on Kaggle has a different audience than a machine learning model deployed internally at a large financial institution to detect fraudulent credit card transactions. As such, the content of your documentation might change accordingly to include more or less detail about specific aspects of the data.

What purpose does the documentation serve?

Similarly, the purpose of the documentation is different for a publicly available dataset when compared to an internally deployed machine learning model, and is different still from a product that provides a chat interface for a large language model. The documentation for a machine learning model used by a financial services company needs to exist for regulatory and auditing purposes, in addition to the typical purpose of remembering how the model works.

How do you provide the documentation?

How you provide the documentation also differs depending on what you need to document. You might be able to add inline comments to a dataset, but then you don’t have a good way to provide an overview of the dataset itself. A machine learning model offers no simple way to include documentation as part of the training output. So far, most standards involve providing a PDF with information, but others such as Hugging Face Dataset Cards, implement a YAML-formatted specification file to publish alongside the dataset on the Hugging Face Hub.

What do you put in the documentation?

When it comes to determining what the documentation should contain, it depends on which component of the machine learning process you’re documenting, and there are a number of standards proposed by academics and prominent players in the data industry.

How to document machine learning components

If you’re deploying machine learning models in your business operations, disseminating the results of data analysis, or integrating machine learning into your product, you need to write documentation. What you write depends on the part of the system that you choose to document.

There are several perspectives to consider:

documenting the datasets
documenting the models
documenting the model results, or output
documenting the system

Documenting the datasets

When you focus on documenting the datasets, you want to capture things like:

How recently the data was collected
From where (digitally and geographically)
How representative the data is against various parameters
Why the dataset was created

For labeled datasets, you want to consider additional components:

Which annotation process was used
How much data was annotated
Which measures you used to validate the annotations
Who annotated the data

Standards for documenting datasets:

Data Cards from Google, which aim to

“provide structured summaries of ML datasets with explanations of processes and rationale that shape the data and describe how the data may be used to train or evaluate models.”
Datasheets for Datasets from Microsoft, which provides a framework of

“questions about dataset motivation, composition, collection, pre-processing, labeling, intended uses, distribution, and maintenance. Crucially, and unlike other tools for meta-data extraction, datasheets are not automated, but are intended to capture information known only to the dataset creators and often lost or forgotten over time.”
Data Statements from University of Washington:, which offers

“a characterization of a dataset that provides context to allow developers and users to better understand how experimental results might generalize, how software might be appropriately deployed, and what biases might be reflected in systems built on the software.”
Data Biography from We All Count, which you can use to record

“a comprehensive background of the conception, birth and life of any dataset … an essential step along the path to equity in data science.”
Dataset Cards from Hugging Face, which

“help users understand the contents of the dataset and give context for how the dataset should be used.”
Dataset Nutrition Labels from MIT and Harvard University:, which offers a

“diagnostic framework that lowers the barrier to standardized data analysis by providing a distilled yet comprehensive overview of dataset “ingredients” before AI model development.”

Documenting the models

In addition to documenting the datasets used for data analysis or machine learning training, you also need to document the models trained on the datasets.

When documenting a machine learning model, you want to capture things like the following:

What data was used to train the model
How the model was trained
How the model features were tuned
Why the model was created
How the model output was tested

For versioned machine learning models, it’s also helpful to include context about what is different between one version of the model and the previous, such as:

Why the new version of the model was created
What training data is different between this version and the previous version

Standards for documenting machine learning models:

Model Cards from Google, which

“provide practical information about models’ performance and limitations in order to help developers make better decisions about what models to use for what purpose and how to deploy them responsibly.”
Data Portraits from Johns Hopkins University, which provide

“artifacts that record training data and allow for downstream inspection”, making it easier for model evaluators to perform tasks like determining if an example output was part of the data input for a model.
Reward Reports from University of California, Berkeley, which are specific to reinforcement learning models and provide a framework for the intended behavior and reinforcement tactics employed for a given reinforcement learning model.

Documenting the model results

It’s important to document the results produced by a model in specific scenarios, which can help you debug and retrain the model as needed. Documentation about model results is often referred to as “explainable AI”, as the goal is to explain the outcomes produced by artificial intelligence (one or more machine learning models, or algorithms).

When you document the model results, you want to collect the following information:

The dataset used to train the machine learning model (and the documentation for the dataset)
The version of the machine learning model (and the documentation for the model, such as the model features)
The input into the trained model
The output of the trained model, in response to the input
The expected output of the trained model, based on the input
How close the expected output was to the actual output, based on the input
Key metrics such as precision, recall, and F1 scores for the model results, especially across different feature segments.
Some possible reasons about why the model produced that particular output

Standards for documenting machine learning model results:

Method Cards from Meta, which

“aim to support developers at multiple stages of the model-development process such as training, testing, and debugging” with prescriptive “instructions on how to develop and deploy a solution or how to handle unexpected situations”
Four Principles of Explainable Artificial Intelligence from NIST, which proposes

“that explainable AI systems deliver accompanying evidence or reasons for outcomes and processes; provide explanations that are understandable to individual users; provide explanations that correctly reflect the system’s process for generating the output; and that a system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output.”
AI Explainability Toolkit from IBM, which provides an extensible toolkit of options to explain AI depending on the audience of the explanation, such as

“data vs. model, directly interpretable vs. post hoc explanation, local vs. global, static vs. interactive”
Standard for XAI from IEEE, which

“defines mandatory and optional requirements and constraints that need to be satisfied for an AI method, algorithm, application or system to be recognized as explainable.”
EUCA: End-User-Centered Explainable AI Framework from Simon Fraser University, which offers

“a prototyping tool to design explainable artificial intelligence for non-technical end-users.”

Documenting the system

It’s important to document the entire system in which a machine learning model operates. A machine learning model is never implemented as a discrete object. Models must be kept updated to avoid drift and thus are implemented as part of a larger system that can include data quality tools, a testing framework, a build and deploy framework, and even other models, such as in the context of an ensemble model.

Documentation about the entire system offers helpful guidance to folks trying to understand a system so that they can maintain, update, debug, and audit the system, to name a few common tasks.

As such, machine learning system documentation needs to include the following:

Details about which tools exist in the system
Details about how updates to the system are performed
Dependencies within the system, including people
Data flows within the system
For models within the system, documentation for those models

Standards for documenting machine learning systems:

System Cards from Meta, to

“provide insight into an AI system’s underlying architecture and help better explain how the AI operates.”
FactSheets from IBM, which provide

“a collection of relevant information (facts) about the creation and deployment of an AI model or service. Facts could range from information about the purpose and criticality of the model, measured characteristics of the dataset, model, or service, or actions taken during the creation and deployment process of the model or service.” with the objective of allowing “a consumer of the model to determine if it is appropriate for their situation.”
DAG Cards from Metaflow, which offer

“a form of documentation encompassing the tenets of a data-centric point of view. We argue that Machine Learning pipelines (rather than models) are the most appropriate level of documentation for many practical use cases, and we share with the community an open implementation to generate cards from code.”

Actually write the documentation

Making sure the documentation actually gets written is the most important aspect of documenting machine learning components and systems.

You can choose to automate portions of the documentation, identify different points of the model development process where it would be prudent to update part of the template that you choose, or whatever works for your team and your processes. You can also define robust accountability mechanisms like checklists.

If you lack processes and accountability, it’s easy to skip writing documentation. All the standards in the world for documenting data-driven systems don’t matter if the documentation never gets written.

Most fields struggle to incorporate documentation into their processes, and build accountability for making sure it gets written, but fast-moving fields like data science and machine learning that are still formalizing their practices struggle even more.

No matter which method you choose for documenting data-driven systems, you must include writing the documentation in your existing workflows.

AI and tech ethics resources

Sun, 02 Apr 2023 10:47:36 +0000

I follow as much discourse around ethics in machine learning, data analysis, and artificial intelligence as I can. These are the resources I’ve used over the years to help me gather knowledge and perspectives and form my own opinions about these types of technology and implementations.

I’ve co-presented two talks on machine learning bias, and gave another on my own about the effects of missing data on data analysis. The slides for those talks can be found on About Sarah.

As an individual person, I have specific interests and biases that inform what content goes from “encountered in my news feed or RSS reader” to “opened in a new tab” to “actually read the thing”. This list is categorized according to those loose interests and resultant taxonomy. Some items might fall into multiple categories.

I’ve denoted things as follows:

📚 for books
📄 for research papers
🎙️ for podcasts
📰 for articles in non-academic publications
💻 for blogs or blog posts
📧 for email newsletters

I’ve also included some content that I haven’t yet made the time to consume, denoted with a ⌛️ emoji.

Dataset creation and curation, including data labeling

📚 Living in Data by Jer Thorp
📚 Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass by Mary L Gray and Siddharth Suri
- I found this book to be pretty dry, and there were some editing issues in the Kindle version that I read, but still highly informative.
📄 Data and its (dis)contents: A survey of dataset development and use in machine learning research
📄 The Data-Production Dispositif
📄 Large image datasets: A pyrrhic win for computer vision?
📄 Datasheets for datasets
📄 An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets
📰 Underpaid Workers Are Being Forced to Train Biased AI on Mechanical Turk (Vice)
📰 The Exploited Labor Behind Artificial Intelligence (Noema)
🎙️ GPT-4 and the Politics of OpaqueAI (ft. Abeba Birhane) by This Machine Kills
- 📄 Discussed in this episode The Values Encoded in Machine Learning Research
🎙️ How Britain Killed its Computing Industry w/ Mar Hicks - Tech Won’t Save Us
- I have a note from listening to this podcast with the phrase “do away with the notion of data as a mirror to society, it distorts society” so relevant if only for that line.
🎙️ How AI Makes Living Labor Undead - This Machine Kills
📚 Atlas of AI by Kate Crawford ⌛️
📚 Data Feminism by Catherine D’Ignazio and Lauren F. Klein ⌛️
📄 Wisdom for the Crowd: Discoursive Power in Annotation Instructions for Computer Vision ⌛️
📄 Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development ⌛️
📄 Documenting Computer Vision Datasets ⌛️
📄 We Haven’t Gone Paperless Yet: Why the Printing Press Can Help Us Understand Data and AI ⌛️
📄 Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data ⌛️
📄 Towards Accountability for Machine Learning Datasets ⌛️

Representation in tech and machine learning, globally, linguistically, racially, or otherwise

📚 Invisible Women: Data Bias in a World Designed for Men by Caroline Criado Perez
- This book is the canonical “missing data = bias” book, but I found it to be problematic for two reasons:
  1. The writing is fairly academic, with citations upon citations and discourse about specific research findings that go on for so long that you can’t quite remember what point the research summary is supporting.
  2. The premise of the book is deeply gender essentialist. A book about missing data that never mentioned gender non-conforming or trans folks, and relied on a definition of gender that was rooted in biological sex. A great reminder to be mindful of the biases present even in books about bias!
📰 An MIT Technology Review Series: AI Colonialism (MIT Technology Review) ⌛️
📄 [2111.15366] AI and the Everything in the Whole Wide World Benchmark ⌛️
📚 More Than a Glitch: Confronting Race, Gender, and Ability Bias in Tech by Meredith Broussard ⌛️
📚 Programmed Inequality by Mar Hicks ⌛️
📚 Black Software by Charlton D. McIlwain ⌛️
📚 Mismatch: How Inclusion Shapes Design by Kat Holmes ⌛️
📄 The Coloniality of Data Work in Latin America ⌛️
📄 Towards decolonising computational sciences ⌛️
📄 The Limits of Global Inclusion in AI Development ⌛️
📄 The Forgotten Margins of AI Ethics ⌛️

Machine learning bias, especially for decision-making

📚 The Alignment Problem by Brian Christian
🎙️ Transcript: Ezra Klein Interviews Brian Christian
📚 Weapons of Math Destruction by Cathy O’Neil
- Much more casually written and full of personal anecdotes than the Alignment Problem, but that probably explains why this book was such a mainstream hit.
📄 Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
📰 Machine Bias — ProPublica (ProPublica)
- Basically the canonical example of irresponsibly deployed machine learning models.
🎙️ The Limitations of ChatGPT with Emily M. Bender and Casey Fiesler — The Radical AI Podcast
📰 Health Care Bias Is Dangerous. But So Are ‘Fairness’ Algorithms | Wired hopped by 12ft.io ⌛️
📰 The Fraud-Detection Business Has a Dirty Secret | WIRED (series) ⌛️
📚 Algorithms of Oppression: How Search Engines Reinforce Racism by Safiya Umoja Noble ⌛️
📚 Automating inequality: how high-tech tools profile, police, and punish the poor by Virginia Eubanks ⌛️
📚 The Black Box Society:The Secret Algorithms That Control Money and Information by Frank Pasquale ⌛️
📚 Predict and Surveil: Data, Discretion, and the Future of Policing by Sarah Brayne ⌛️
📚 Artificial Unintelligence: How Computers Misunderstand the World by Meredith Broussard ⌛️
📚 Technically Wrong: Sexist Apps, Biased Algorithms, and Other Threats of Toxic Tech by Sara Wachter-Boettcher ⌛️
📚 Hello World by Hannah Fry ⌛️
📚 Your Computer Is on Fire edited by Thomas S. Mullaney, Benjamin Peters, and Mar Hicks ⌛️
📚 Captivating Technology: Race, Carceral Technoscience, and Liberatory Imagination in Everyday Life edited by Ruha Benjamin ⌛️
📚 Sorting Things Out: Classification and Its Consequences by Geoffrey C. Bowker and Susan Leigh Star ⌛️

Machine learning development and implementation

📚 The Ethical Algorithm by Michael Kearns and Aaron Roth
- A practical discussion of ethical approaches to algorithm design and the involved tradeoffs.
🎙️ Deploying Machine Learning Models Safely and Systematically – The Data Exchange
📄 On the Dangers of Stochastic Parrots
🎙️ Emily M. Bender — Language Models and Linguistics | gradient-dissent – Weights & Biases
🎙️ The Power of Linguistics: Unpacking Natural Language Processing Ethics with Emily M. Bender
📚 Understand, Manage, and Prevent Algorithmic Bias: A Guide for Business Users and Data Scientists by Tobias Baer ⌛️
📚 The Oxford Handbook of Ethics of AI edited by Markus D. Dubber, Frank Pasquale, and Sunit Das ⌛️
📚 You Look Like A Thing and I Love You by Janelle Shane ⌛️
- I’m halfway through this and it’s excellent so far. A very straightforward discussion of how ML works, how it doesn’t work, and why.
📄 The Fallacy of AI Functionality ⌛️

Auditing, testing, and monitoring machine learning

Alternate approaches to data-driven and ML-driven systems

General resources

🎙️ The Radical AI Podcast
- A podcast at the intersection of ethics and artificial intelligence.
🎙️ The Data Exchange Podcast
- A podcast from folks in the data industry.
🎙️ This Machine Kills
- A podcast from researchers and critics of AI and the data industry.
🎙️ Tech Won’t Save Us
- A podcast that critically examines the tech industry, data included.
🎙️ Gradient Dissent from Weights and Biases
- A podcast from folks in the data industry.
📧 The AI Ethics Brief
💻 ★❤✰ Vicki Boykis ★❤✰
- An excellent blog from an ML practitioner and champion of data grunt work.
📧 Import AI
- Not about ethics or bias specifically, but a worthwhile perspective from an AI insider.
📧 AI Snake Oil
📧 Counting Stuff
📧 Benn Stancil
📧 Rebel Tech
📄 arXiv > Computer Science > Artificial Intelligence
📄 arXiv > Computer Science > Computer Vision and Pattern Recognition
📄 arXiv > Computer Science > Computers and Society
📄 arXiv > Computer Science > Computation and Language
📄 arXiv > Computer Science > Machine Learning
📄 arXiv > Computer Science > Human-Computer Interaction
- I use a keyword filter on the Computer Science portion of arXiv with my RSS feed reader to pull up potentially relevant articles, but these are the areas in which they usually appear.
📄 AAAI/ACM Conference on AI, Ethics, and Society (AIES)
📄 ACM Conference on Fairness, Accountability, and Transparency (FAccT)
- Useful conference proceedings to dig into each year.

The value in research gaps

Mon, 20 Mar 2023 10:57:21 +0000

There’s value in the holes. If you search for information about a topic and don’t find very much about it, it can be a clear signal that more research is necessary or desired to find the answers.

A research gap is exactly how Dr. Tina Lasisi, interviewed for the Melaninology episode of the Ologies podcast, describes how she got into this field of research [starting around 6:22]:

I have always been aware that different people have different skin color, but I never thought about how it was patterned around the world. What about other traits? How do those vary, and how do those evolve? And my immediate question as a Black woman, was okay what about my hair? Like okay, I understand why my skin is brown, but why is my hair curly? And the wild thing is there wasn’t a good answer! What should’ve been a really quick Wikipedia search that satisfied my curiosity, became this rabbit hole where I basically had this Postdoctoral Fellow who was at our college who took me under his wing and was like hey, let’s talk about BioAnth [Biological Anthropology], and I was like, oh, so I have all these questions and I can’t find anything about like, hair, and he basically was like well, sounds like that could be something for like your undergraduate thesis! And like as an undergraduate I decided ok let me get hair samples and measure them, and like yeah, long story short, basically this thing that should’ve been a short Wikipedia search ended up being a decade plus journey into understanding this trait and like why humans have it. (transcription mine)

I’d wager many other academics have a similar origin story for their own research projects.

Unfortunately, if you rely on a large language model to do research for you, or help give you ideas about what to research, the nature of the tool means that instead acknowledging that there are no results for your prompt, you instead can get output full of “best guess” citations based on words that are semantically similar to the words that you prompted.

Ben Davis writes about this phenomenon for Artnet in We Asked ChatGPT About Art Theory. It Led Us Down a Rabbit Hole So Perplexing We Had to Ask Hal Foster for a Reality Check. He and his colleagues found that if you attempt to perform research with ChatGPT, you are instead likely to receive a list of nonexistent citations:

Sometimes—if, for instance, I ask it to “give me a list of citations about the influence of Artificial Intelligence on European Medieval Art”—it accurately tells me that this query makes no sense—but then provides a list of made-up references anyway

Ben makes several attempts to get the system to provide references about a topic that likely has very little, if anything written about it, going so far as to specifically ask for citations that actually exist:

When I ask a follow-up, specifying that the references now be actually “real,” my chatbot helper is again very helpful, but again just makes stuff up

Ultimately, he concludes that this flaw is not a surprise, and is fact completely expected due to what ChatGPT is built on:

The glitch seems to be a linear consequence of the fact that so-called Large-Language Models are about predicting what sounds right, based on its huge data sets. As a commenter put it in an already-months-old post about the fake citations problem: “It’s a language model, and not a knowledge model.”

In other words, this is an application for sounding like an expert, not for being an expert—which is just so, so emblematic of our whole moment, right?

This flaw makes large language models a decent application for things like thought leadership, a common forum for sounding like an expert. But for circumstances where it truly matters to be an expert—like academia, or writing documentation—it’s a serious problem.

It’s one thing to confabulate thought leadership — it’s quite another to lie about product functionality in contractually offered software documentation.

There’s value in identifying the holes in knowledge and understanding. If you gloss over knowledge gaps with generated drivel, you miss out on the opportunity to dig deeper to learn about something—and so too does anyone that might discover what you learned in your research.

Considering types of meetings

Fri, 17 Mar 2023 08:22:55 +0000

The next time you’re in a meeting, wishing you weren’t, you might want to consider why the meeting feels so insufferable.

Often it’s because no one has bothered to consider the purpose of the meeting.

Cam Daigle devised a classification system that provides an excellent framework for improving meetings. They declare that There are three types of meetings:

I believe all meetings can either be defined as either Status meetings, Feedback meetings, or Decision meetings.

For each type of meeting, Cam identifies the goal, appropriate scale, power dynamic, and risks inherent in each type. For example, for status meetings:

The goal of a Status Meeting should be to disseminate information. That’s it. If the attendees of the meeting come away with current and relevant information about whatever the hell the meeting was intending to communicate, it’s done its job.

My favorite meeting type is the feedback meeting—mostly because meetings of this type are the most frequent devolution state of a status meeting or a decision meeting (in my experience):

Everyone is in a room talking about what they accomplished in the last 24 hours, and then a few folks start offering feedback on a specific approach to solving a problem that someone took. A status meeting just became a feedback meeting.
Or maybe you’re in a meeting room, trying to decide whether to build something based on a given design, and you get an hour of feedback on the design but no clear path forward. A decision meeting became a feedback meeting.

For a feedback meeting, Cam is careful to identify 2 crucial devolution states, of which the second is more critical:

Be extremely careful about decisionmaking.

The moment the presenter stops receiving feedback and starts reacting to it, the meeting dynamic is at risk.

That’s right, a feedback meeting can also devolve into a decision meeting.

I highly recommend reading Cam’s entire post. Since doing so, every meeting I go to and thinkpiece about meetings that I read has me sitting there wondering what type of meeting I’m in, or is being discussed.

Is my 1:1 with my manager a decision meeting or a status meeting?
Is this product review a feedback meeting or a decision meeting?

The next time you schedule a meeting, give it an agenda, but also consider which type of meeting it is—and whether it should be happening at all.

Chat apps are no substitute for documentation

Wed, 15 Mar 2023 09:25:14 +0000

As a technical writer, I have a mixed opinion of chat applications like Discord and Slack. On the one hand, they make it easy to quickly get ahold of someone who can answer your questions, which is a relief if you’re struggling to gather information you need to write a draft.

On the other hand, because it’s easy to quickly get ahold of someone who can answer your questions, that convenience can implicitly incentivize folks to neglect documentation. This is true on both sides:

Folks with questions might ignore documentation that exists because it seems easier or faster to just ask in the chat app.
Folks answering questions might prefer not to spend lots of time writing system design documentation, detailing the decisions they made and why, when they can just answer those questions as needed.

Jason Scott shares this mixed opinion. As he points out in Discord, or the Death of Lore:

“I have no disputes as the popularity of the places, the things that happen there, and the unquestioned vivaciousness of being the party that never seems to end and everyone wants to join.

I just happen to be the sort of person who notices there’s no decent fire exits and most of the structure is wood and there’s an… awful lot of pyrotechnics being set off.”

I feel the same way.

I wonder if the folks who have their focus time interrupted by detailed, somewhat archaic questions that require a backstory, ever wish they had documented the answer so that they could respond with a link and move on with their day.

Chat apps like Discord end up diluting the available knowledge because the content shared in them isn’t persistent, and the allure of an always-available answer breaks down when the person that could answer is no longer available. Jason refers to this as the “lore-to-knowledge transfer”:

The danger in this process, the potential lost ballast in the rise to the skies, is that the lore-to-knowledge transfer is lossy, messy, and arbitrary. Maybe those in the know want to keep the information to themselves, so it won’t be given to whoever the person or persons are who are laying down the written form. Maybe the chronicler of information has blind spots they don’t know about and not enough people to correct them. Or, more likely, you have to set the “noise filter” of the information to not go down the rabbit and rat holes of contingencies that maybe a dozen or two people will even want to know about, to the favor of that which everyone will need. The outcome is always the same: Lore loses in the long run.

The process of documenting information can break down silos (information is more available), expose blind spots in an approach (something is missing here), and enable asynchronous knowledge transfer (documentation is always online).

Etsy blogged about an interesting approach five years ago, Etsy’s experiment with immutable documentation, where they built a plugin that engineers could use within Slack to update or create documentation:

At Etsy we’ve developed a system for adding how-docs directly from Slack. It’s called “FYI”. The purpose of FYI is to make documenting tactical details – commands to run, syntax details, little helpful tidbits – as frictionless as possible.

I’d be curious to learn whether they’re still using this system, and how it has aged.

Measuring data (and documentation) quality is hard

Mon, 13 Mar 2023 10:24:27 +0000

Gwen Windflower asks Are you actually measuring data quality?:

Data is a woven net thrown over the world, capturing it in a grid of variable resolution. We want to capture as much detail as we possibly can, but it’s then crucial to make judicious decisions about how we translate what we capture into a useful map.

Documentation involves a similar discernment process as data analysis and data engineering—given a large set of inputs, identify how to make sense of the output and where to start first. Defining priorities and classifying information is vital.

When it comes to quality in particular, and how to measure quality, Gwen addresses the common practices of data analysts:

Typically though, when it comes to quality, we proceed to measure things like the internal consistency of multiple printings of our map, the resolution of the ink, and how quickly we print new maps in response to changes in the terrain. Two related maps stay internally consistent? Quality. Granular resolution of detail? High quality. Map updates in near-realtime? The highest quality. We often assume that making these kinds of measures higher in our output is an unalloyed good.

Again, this is a common practice in documentation. It’s tempting to align documentation quality measures with documentation coverage (parity with product functionality), documentation freshness (updated quickly), or with particular measures about the writing quality itself, such as sentence length, page length, or word complexity.

But as Gwen points out, measuring quality in these ways is flawed:

There’s a problem with these self-referential, almost tautological measures though: none of them tells us if a map is fulfilling its purpose.

Maps should be measured on how well they get you where you want to go. To do that maps need to be as detailed and accurate as necessary and no more.

If you want to measure the quality of your output, you also need to assess its usefulness.

To measure quality, we need to look beyond the data (or the documentation) itself, and look at the function. Gwen makes this clear, continuing with the map metaphor:

to know the proper level of detail, pace, and presence for our maps, we need to know where we want to go.

If you want to hike through a national park, you want a trail map, not a road map. Similarly, you need to understand why you’re creating, refining, or analyzing data so that you can produce something that is high quality for the purpose.

The payroll department might have an objective to improve payroll quality, measured by reducing paycheck errors across the company from 5% to 1%. If you’re managing their data pipeline, knowing what the data is used for and what the quality issues are can make a big difference in how you approach your work. You might start by improving the accuracy and reliability of the data before you worry about reducing latency of the data to be near real-time.

Similarly, documentation is a supporting team. We do our own work, but it’s relatively useless if it doesn’t help folks meet their objectives. To write high quality documentation, we must consider the goals of our audience, and what they want to accomplish with the product, to provide relevant and useful documentation.

As a supporting team, data teams face a challenge that Gwen identifies:

There is constant discussion of how hard it is to measure the impact of data teams, and that’s directly because of a lack of intentionality with our data.

That’s something that applies to tech writing teams. The impact of a data team, and the impact of a documentation team, can be measured by identifying the objectives that those teams are helping others meet. Who does our work support, and did we do our work with the goals of that audience in mind?

Music trends and data errors: 2022 in music

Sat, 31 Dec 2022 14:40:47 +0000

In 2022, I had no true “obsessions” in my music listening, unlike last year. Instead of any standout artists, I flitted from artist to artist as they released new albums or other things prompted me to rediscover how much I enjoyed their music.

This was a year for breadth, rather than depth, and also for discovering the limits of my music data collection mechanisms.

If you want to skip around, here’s a table of contents:

Top artists of 2022

My top artists of 2022 had a lot of familiar names.

Rather than any one consistent artist, the real pattern was one of shifting obsessions. Let’s zoom in.

February: Caribou

When 2022 kicked off, I barely listened to Caribou. I listened to 3 songs in January. In February, I went to their concert on a Wednesday at the Fox Theater.

Their live show blew me out of the water. I was entranced, captive to the flow of the music as they played Sun for what felt like twice as long as the studio track duration. In an attempt to recapture that feeling, I listened to Caribou a lot in the weeks after the show, and as a result they ended up one of my most-consistently listened to artists of 2022.

June: HAAi

I discovered HAAi on October 08, 2018, listening to her track Be Good. I have no recollection of this, but the track is familiar to me. I didn’t listen to any new tracks by her until 2020, but the next one I heard, FEELS, was one I’ve since listened to 26 times.

Nevertheless, when I heard that she was coming out with a debut album, and it was produced with Jon Hopkins, I was thrilled.

The audio for her single from the eponymous album, Baby We’re Ascending, was released on May 4, 2022, and the music video was released on June 1st.

HAAi’s debut album came out on May 27, 2022, and on June 1st and June 2nd, I listened to the whole album all the way through.

I added her album to my library on June 8, 2022, and on June 18th, I listened to HAAi tracks 34 times. Judging by the frequency of listens on that day, there was also a data error afflicting those results, but I listened to the album all the way through at least once that day.

September and October: TSHA

Not as prominent as other months and other obsessions, I still spent a lot of time listening to TSHA in September and October this year.

TSHA is another artist that I listened to quite a bit over the last couple of years, but didn’t yet have a full album out. She released her highly anticipated (at least by me) debut album Capricorn Sun in October, and it quickly went into strong rotation for me.

Three of my top 10 songs of the year are TSHA tracks, and she was one of the artists that I listened to at least once every month this year. I finally saw her DJ for a bit at Portola Music Festival, but I can’t wait to see her throw down for a headlining set.

November: Frightened Rabbit

Frightened Rabbit is a band I’ve listened to for a long time.

The first song of theirs that I added to my iTunes library was Last Tango in Brooklyn, off a demo album, on October 1, 2009. That was also the first track I heard of theirs, on Sunday, November 15, 2009. However, I haven’t listened to them in a while.

So what prompted me to start listening to them again in November, and with such fervor?

I updated my podcast feed.

A friend of mine had recommended the Object of Sound podcast, with Hanif Abdurraqib, a poet and essayist that he was a fan of. I’d added the podcast to my feed but hadn’t yet listened to it, and in early November I updated my podcast feed and saw the episode for When It’s All Gone, Something Carries On (A Tribute to Scott Hutchison).

That episode was released on November 4, 2022, and seeing that podcast in my feed, prompted me to revisit their music.

Subsequently, I listened to Frightened Rabbit intensely throughout November. I listened to Spotify albums downloaded offline for the entire 6+ hour flight back from my Thanksgiving travels, so that isn’t reflected in the data for November 29.

I finally listened to the podcast on the bus to the airport on November 29, and it moved me so much that I cried.

I kept up the intensity of my listening for a few days into December, but since then, the intensity of my listening has dwindled.

For reference and comparison, I saw Frightened Rabbit live in 2013, long before their frontman died, and my post-concert peak listening was 13 times in one day.

All of those days of listens are overshadowed by my listening patterns in November and December this year.

Another mini obsession: Hadiya George

Beyond those three notable artist-of-the-month interludes, I had another intense listening stretch that wasn’t notable enough to make it to the top 10 artists, but did crack the top 10 songs.

Hadiya George’s track Hot Flavor, remixed by Godmode Smash Brothers, was one of my top tracks of the year.

I discovered the track on August 21, 2022, and then listened to it 14 times the next day. Over the next couple of months I listened to that track, or the extended remix, once a day on 4 occasions. And that’s it!

As far as I can discern, I discovered the track on SoundCloud, liking it and reposting it the first day I heard it. August 21st is a Sunday, and SoundCloud Weekly refreshes late on Sunday night for me, so presumably that’s where I heard it for the first time.

It’s also possible that the track ended up in the autoplay recommendations when I listened to Carly Rae Jepsen’s track Beach House at the suggestion of a friend. I find that SoundCloud’s autoplay is pretty unmatched, at least for my music taste.

I listened to the track mostly on YouTube, because that’s the only place I could find the track, forgetting that I had discovered it on SoundCloud.

Eventually I purchased the extended version, and that’s where the listens at the end of September came from.

Consistent favorites

In addition to the moments when I dug deep into an artist’s catalog, I had some standby favorites of the year.

salute

The producer and artist salute was one of my favorite artists of 2022. My eighth-most-listened-to artist of the year, those listens were concentrated on just 15 tracks, making him an outlier in terms of intensity of my listening activity for the top 10 artists.

In addition to making an appearance on my top 10 artists, he also made an appearance on my top songs, with his track Honey being my second-most-listened-to track of the year. He doesn’t have an album out yet, otherwise he might have shown up there as well.

I didn’t listen to salute every month of the year, but I enjoyed his music all the same.

I first discovered him in June 05, 2017 15:28:03, with a track called Weigh it Up (featuring Krrum). I have to imagine it was on Spotify, because while salute is great, this track isn’t really my style. Spotify put another track from salute on my Discover Weekly the next month, his track Light Up, which has a similar high production vibe that is more typical of standard EDM.

The first track of his to really kick, for me, was Want U There, which I first heard last summer June 29, 2021 10:35:05, which has a much more UK garage backbeat and vibe.

Fred Again..

My top artist of last year, Fred was also my top artist this year—but by a much slimmer margin than last year.

He was one of six artists that I listened to every month this year, joining Caribou, DJ Seinfeld, Logic1000, TSHA, and warner case.

On the other hand, I only saw him live once despite him playing San Francisco 3 times this year. I was underwhelmed by his first live appearance the weekend before Coachella, despite being stoked enough to show up early and stake out a front row spot at Great American Music Hall.

Having 3 albums out certainly helped him beat out the other contenders for top artist in terms of sheer output—the 193 total listens for this year are spread across 69 different tracks.

Most of my listening activity was concentrated in September and October, when his third album, Actual Life 3, was released.

Logic1000

Logic1000 making a stealth top 10 appearance. I didn’t consciously listen to her this year, but she squeaked in as one of my most consistently listened to artists of the year as well.

She came out with a few new tracks this year that I enjoyed, notably Rush and Can’t Stop Thinking About.

I also added 2 of my favorite songs of hers, YourLove and Safe in My Arms, to my library this year. Given that I moved away from listening to music on Spotify this year, that helped as well.

She joins salute in the list of artists that I’m hoping releases an album soon, but in the meantime I’m enjoying her singles.

I had plans to see her play earlier this year, but she cancelled her entire tour hours before I was supposed to see her. I hope she’s well.

Listening habits over time

As I mentioned in my Spotify Wrapped post this year, I made a concerted effort to diversify where I was listening to music. I took some time off from working this year, and that also contributed to changes in my listening behavior this year.

In general, the amount of time I spent listening was greatly reduced this year, most notably removed from the workday.

Overall, my daily listening habits flattened, no longer spiking on the weekdays while I filled my time with music while working.

While my overall listening volume is lower, you can also see the dip mid-year, when I’m working less. That coincided with moving in with my partner, a pattern that in the past has led me to listen to less music in general.

Where did my money go?

This year, I wanted to put some more effort into figuring out who gets paid when I’m listening to music. It’s still a rough estimate, since I don’t know how much of my listening activity happened in which platform, but it’s something.

If I assume a $1 average price paid per song, this chart shows the amount that artists earned for purchases that I made on Bandcamp, and for purchases that I made on the iTunes store.

While Bandcamp is extremely transparent about how much money they pass on to artists, every other platform is more opaque and steeped in layers of “it depends”. For the purchase calculations, I’m assuming 82% of the price is passed on to the artists when purchased from Bandcamp.

For iTunes, I’m assuming that 70% of the payment is passed to artists when I buy a track from the iTunes store. I relied on an article from The Guardian How much do musicians really make from Spotify, iTunes and YouTube?, published in 2015, for the iTunes rate details.

In an attempt to determine how much money artists earned from my streaming behavior, I charted out the amount artists would earn if all of my listens for the year occurred on a given service.

For the royalty rates from streaming services, I relied on the Streaming Royalty Calculator from Omari MC, which gave the following rates:

0.00437 for Spotify
0.00069 for YouTube
0.003275 for SoundCloud

All streaming rates end up being approximations, because the actual payout depends on the revenue for the streaming service, the share of an artist’s listens compared to total listens at the streaming service, as well as any negotiations made by the artist’s record label with the streaming service, if applicable.

While streaming revenue is recurring revenue, it’s wildly lower than the revenue that artists get when someone buys their music. Given that I still use streaming services, I think it’s a great idea to continue purchasing music.

Data errors and my top song of the year

When I first went through this analysis of my music habits, I discovered that my top song of the year was Quiet Little Voices by We Were Promised Jetpacks.

This was a bit of a surprise, because while I love that song, it isn’t one I had a strong memory of listening to frequently, let alone more than any other song of the year. So I dug a bit deeper, and that’s where things started to fall apart.

It turned out that all the scrobbles for that track were on June 8, 2022. On that Wednesday morning, I took the bus to an appointment, walked to the gym for a workout, then took the bus home.

According to my data, while I did that I listened exclusively to Quiet Little Voices on repeat from 4:30AM until 8:20AM, and then I rotated Fred Again..’s track Eazi (Do it Now) and Jamie xx’s track I Know There’s Gonna Be (Good Times) [feat. Young Thug and Popcaan] into the mix from 8:30AM until 10:00AM.

Starting at 11:00AM until 2:30PM I followed a similar pattern of listening to 7 different songs at impossible frequencies.

Why was this impossible?

In order to count a track as a listen (scrobble, in Last.fm parlance), the following needs to be true ¹:

The track must be longer than 30 seconds.

And the track has been played for at least half its duration, or for 4 minutes (whichever occurs earlier.)

That means that according to the data, I was sometimes listening to as many as 5 songs in a 5-minute period. However, the shortest song I was ostensibly listening to is nearly 3 minutes long, and most tracks were at least 4 minutes or longer ².

For those songs, I had a number of impossible track frequencies throughout the time period in the morning, and the second time period in the afternoon:

track_name	listens
Quiet Little Voices	73
Louder (Lefti Remix)	35
Eazi (DoItNow)	23
Mbiffé	20
Hell No	15
I Know There’s Gonna Be (Good Times) [feat. Young Thug & Popcaan]	13
Never Come Back	12
Start Again	7
Inyani Feat.Oluhle & Aaaron	3
O’Flynn - SGD (Soundbwoy Killah Remix)	3
Sorry (Greene Edit)	3
exe.cute	3
Mir a nero (Original Mix)	2
For Sarah (Live DJ Mix)	1

After I discovered this, I identified and excluded the data in Splunk ³ so that I could choose on a search-by-search basis whether to exclude this anomalous day from my data.

I thought that was the end of it — discovered a specific anomaly in the data on one day, remove that day from the data, and move on.

More errors than I realized

A month later, I discovered that the problems went deeper than I thought. There were more errors beyond just this one day. As part of analyzing my music data to compare it with Spotify Wrapped, I determined whether I was habitually more loyal to songs (listening to the same songs frequently in a given time period) or sought out variety.

To do this, I collected my listening behavior into 20 minute bins and evaluated the number of repeated tracks in each bin, then counted the number of bins with repeated tracks (loyal) and different tracks (variety):

`lastfm` |  eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false")
    | search exclude=false
    | bin _time span=20m
| stats count by track_name,_time
| eval habit=if(count>1, "loyal", "variety")
| stats count by habit

After separating my listening habits into those 20 minute bins and assessing loyalty compared to variety, I wanted to know which tracks I was ostensibly loyal to:

`lastfm`
| eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false") | search exclude=false
| bin _time span=20m
| stats count by track_name,_time
| eval habit=if(count>1, "loyal", "variety")
| search habit=loyal
| sort -count, _time

And this is where the data errors show up. A lot of my “most loyal” tracks were seemingly double or triple scrobble submissions, almost certainly by the mobile scrobbling app, based on the times and days when the data collection errors occurred. I did a deep dive into why this might be happening ⁴, but I still wanted to see if I could consistently identify the data errors.

Based on an average track length of about 4 minutes, I first isolated the number of 20-minute bins that had more than 5 songs in them.

`lastfm`
| bin _time span=20m
| stats count, list(track_name) as tracks by_time
| search count >5
| sort -count, _time

There were a number of legitimate-seeming track patterns in those bins, so I narrowed it further to isolate the more improbable time periods wherein I listened to at least 7 songs in 20 minutes:

`lastfm`
| bin _time span=20m
| eval year = strftime(_time,"%Y")
| stats count, list(track_name) as tracks list(artist) as artist by _time, year
| search count >7
| sort -count,_time

I figured that any instances of 7 tracks scrobbled in a 20-minute period is almost certainly a data error. Given that most tracks I listen to are an average of 4 minutes long, 7 tracks in 20 minutes would mean that most tracks were 3 minutes each or less.

If I use an improbably high estimate of 8 songs per 20-minute bin, I can track down the worst of the worst bins over the years. If I focus on the last few years specifically, there’s quite a lot of issues:

year	count
2020	61
2021	84
2022	24

Oh no. Let’s see what the worst bin offenders were:

2021-11-27 17:20:00 I ostensibly listened to 33 Fred again.. tracks. I was on an airplane, so it’s possible that I really did listen to all these tracks — just not in the same 20-minute period.
2021-11-27 17:00:00 has 28 songs recorded, also all by Fred Again.., 20 minutes earlier.

Looking at just the bins from 2022, the worst ones are part of the time period I identified on June 8:

2022-06-08 9:00:00 I listened to 17 tracks.
2022-06-08 13:40:00 I listened to another 17 tracks.

This is disheartening, but also a relief that for this year, I’ve excluded the most egregious outliers in my analysis.

Can I trust the data?

Back to the topic at hand—why am I digging into these data anomalies and errors at all? Because I use my Last.fm data to prop up my shoddy memory, I rely on it to reinforce or remind me about my favorite songs in a year.

If I can identify issues with the data based on statistical anomalies, and better yet, identify how the data is created so that I can trust my data. If I can understand how the data is created, I can improve its quality going forward, and clean up past data errors as well.

Last.fm recognizes the importance of high-quality data, and filters API requests sent to the scrobbling endpoint. Some relevant message codes:

3 : Timestamp too far in the past.
4 : Timestamp too far in the future.
5 : Max daily scrobbles exceeded.

They recognize that there are logical improbabilities in the data that would indicate errors upstream, and throw that data out.

In the case of my music data, I have two or three sources for listening data, or play count data, for a track:

Last.fm
iTunes metadata
Spotify metadata

Each of these records a listen, or a play, slightly differently.

Last.fm counts a scrobble for a song if it’s been played for at least half its duration, or for 4 minutes ¹.
iTunes play count is muddied with the Apple Music streaming service. Apple Music follows the same pattern as Spotify ⁵, but iTunes seemingly requires a song to be played in its entirety before it is counted as played for the play count ⁶.
Spotify counts a stream for a song after someone listens for at least 30 seconds ⁷.

I don’t have any single ground truth measure to validate the data in Last.fm, but I can cobble something together from iTunes.

If I return to the first outlier track that led me down this rabbit hole, I can see that the overall play count for We Were Promised Jetpacks’ track Quiet Little Voices according to iTunes is 73. However, Last.fm lists the plays for that track for just this year alone as 74 plays, with 136 total plays.

This track has been in my iTunes library since 2009, so let’s go back in time. If I refer to a Library.XML file for my iTunes library from 2016, I can see that the play count for that track was 61:

    <key>Play Count</key><integer>61</integer>

That’s more listens than Last.fm was aware of through that time, but Last.fm’s ability to scrobble music was much more rudimentary then. The bar to count something as a play in iTunes is also much higher than the threshold to count a track as scrobbled.

What caused this data error?

There are patterns in the data discrepancies, or some likely attributions that I can come up with, like:

scrobbling from mobile when it’s been awhile since I scanned for scrobbles using the app.
scrobbling after a prolonged offline period of time, such as a plane flight
listening on multiple data streams at once or overlapping, such as listening to Apple Music and Spotify at the same time, or the Spotify app and Spotify website at the same time.
discrepancies with how developers that send data to Last.fm record a listen of a track, such as by scrobbling a track before the Last.fm definition of a scrobble is reached ⁸, which could explain another data bin that I discovered ⁹.

Ultimately, it feels like the more I dig, the more it feels like the listening data I have is like the points on the improvisational comedy show Whose Line is it Anyway: made up, and they don’t matter.

I track my music data for fun. From a data analysis perspective, it’s a challenge that there is no single source of truth that can provide “ground truth” accuracy for my music listening data, but it’s a great reminder that the same is true for most data sources in the world.

We rely on imperfect data collection methods to identify patterns and draw conclusions. It’s only by having a deep understanding of how the data came to exist, and audits throughout the analysis process, that we can be confident in the results of the data analysis.

2022 in music, and what’s in store for 2023?

2022 was a unique year in music listening. My listening volume was down, my listening habits were spread across a lot of different services, but I still spent time with artists whose music I treasure. Here’s to more UK garage in 2023, some new albums from upcoming artists, and even more discoveries.

When is a scrobble a scrobble? in the Last.fm API documentation. ↩︎ ↩︎

The following table outlines the tracks and durations that I listened to during that time period:

track_name	artist	duration
Quiet Little Voices	We Were Promised Jetpacks	4:21
Louder (Lefti Remix)	Crush Club	4:07
Eazi (DoItNow)	Fred again..	3:37
Mbiffé	Lokua Kanza	4:02
Hell No	Ingrid Michaelson	2:55
I Know There’s Gonna Be (Good Times) [feat. Young Thug & Popcaan]	Jamie xx	3:34
Never Come Back	Caribou	5:05
Start Again	Kidnap	4:00
Inyani Feat.Oluhle & Aaaron	Re.You	7:15
O’Flynn - SGD (Soundbwoy Killah Remix)	Hundred Flowers Records	7:36
Sorry (Greene Edit)	Jacques Greene	6:00
exe.cute	Marc DePulse	6:55
Mir a nero (Original Mix)	Michel Cleis	12:15
For Sarah (Live DJ Mix)	Tourist	4:48

↩︎

I wrote the following evaluation statement, and then searched for the data that didn’t match it:
```
| eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false")
| search exclude=false
```
This approach meant I could identify the erroneous data without automatically excluding it. ↩︎
I dug into alternate scrobbling apps, in case that was the issue, and discovered one called Eavescrob, which has an FAQ that includes the following note:

Due to the limitation of iOS system, iOS only keeps the last time point of a song you’ve played, so currently the repeated plays are timestamped based on your last played date. Let’s hope a better solution would come to iOS eventually.

And thus I had a kernel of information to attempt to track down. In an attempt to locate the Apple Music API endpoint or framework this information came from, I dug into the open source code for Finale, another Last.fm scrobbling app. I couldn’t find any leads, so I reached out to the developer on Twitter, and he was kind enough to point me to another repo, where he has the code for retrieving the information.

It turns out that the Apple Music API endpoint, get recently played tracks, is not the main source of information. Instead, it’s a Media Player framework which permits queries of specific media items that have been played. That lastPlayedDate variable confirms that the data that is collected is, in fact, the most recent play date for the item (rather than an array with the history). You can perform a query with any elements of an MPMediaItem. In the case of the Finale scrobbling app, the developer queries the songs and then filters them for those where the lastPlayedDate is after a specific time.

It’s always a challenge to use an API or a framework in a somewhat adjacent manner to which it has been intended, and this is likely a case where, if the framework had been designed to communicate all playback activity, that would be possible. But in the context of, say, writing a shuffle algorithm, and meeting more clear and internal product needs, the framework would only need to know how recently the track had been played — not the entire track playback history.

And so we hang out in the niche and acquire subpar data for our purposes.

Presuming that the Last.fm scrobbling app for iOS works the same way as the Finale app, or at least using the same framework, an issue with cached data, a malformed query, something else, or some combination of all of those that likely caused the data corruption on June 8 and other dates. ↩︎
“Plays are recorded when a user initiates song playback in Apple Music for more than 30 seconds.”, according to Understand your analytics in the Apple Music for Artists documentation. ↩︎
This is according to a discussion on Reddit, and another on StackExchange. The conversation on StackExchange from 2011 about What does “plays” really count in iTunes? points out:

play count == number of times the file played right to the very end.

The conversation on Reddit on the Apple Music subreddit, What counts as a “play count”? from 3 years ago affirms that: listening “Up to less than :10 of the song ending.” is what counts as a play in iTunes.

The official documentation for playCount in the Media Player framework and the MusicKit Library documentation is, as anticipated, vague and useless:

The number of times the user plays the media item.

or alternately:

The number of times the user played the song.

In my own experience, tested while writing this, I can confirm that the denizens of the internet forums are correct, and the play count on Apple Music, formerly known as iTunes, increments only when a song has been played in its entirety. ↩︎
“Song stream: Counted when someone listens for 30 seconds or more”, according to How we count streams in the Spotify for Artists documentation. ↩︎
See the earlier definition of what a scrobble is, according to Last.fm. It occurred to me as well that since Last.fm will send the “currently playing” track as well as the history of tracks played, that it could be an anomaly there, but I discard the “currently playing” events in the Last.fm Add-on for Splunk configuration, and I further deduplicate the data at search time by unix timestamp (uts) so if the timestamp matched exactly, I wouldn’t count duplicates. ↩︎
That last explanation is probably the case with a 20-minute bin from earlier this month, 2022-12-04 23:20:00, where I listened to these 8 tracks in a row:
- Orbiting
- Can’t Go Back Now
- Takes so Long
- Antarctica
- Wish I Could Forget
- Thunder
- Hideaway
- Orbiting
Many or most of these songs are under 3 minutes long, which can explain why they nearly fit in a 20 minute time block:

track_name duration

Orbiting 2:55

Can’t Go Back Now 2:18

Takes so Long 3:07

Antarctica 3:15

Wish I Could Forget 2:57

Thunder 3:24

Hideaway 2:47

All of these songs are by The Weepies, except for Thunder, which is a track by Imagine Dragons. I must’ve been playing Heardle in the middle of listening to a bunch of mopey folk songs by The Weepies, and listened to that on Spotify, which has a lower threshold for counting something as played than iTunes does. ↩︎

track_name	duration
Orbiting	2:55
Can’t Go Back Now	2:18
Takes so Long	3:07
Antarctica	3:15
Wish I Could Forget	2:57
Thunder	3:24
Hideaway	2:47

Spotify Wrapped 2022: My listening personality and more

Tue, 06 Dec 2022 10:59:35 +0000

It’s time for another deep dive into this year’s Spotify Wrapped! I’ve been doing this for almost as long as Spotify Wrapped has existed. Check out the past years’ posts: 2018, 2019, 2020, 2021.

I collect my music data using Last.fm, and then I wrote a custom Last.fm add-on for Splunk to send my listening data to Splunk. I then use the app I developed, Music App for Splunk to visualize the results. Both the add-on and the app are open source, so feel free to try it out yourself if you use Last.fm.

Spotify vs Last.fm

In the four years that I’ve been writing these posts I’ve reveled in the data missing from Spotify Wrapped and crow a bit about the superiority of quantifying my music habits more holistically with Last.fm. Spotify Wrapped, as you might know, is based on your listening patterns in Spotify from January 1st through October 31.

This year, I made more of an effort to diversify my music listening habits and also spent less time on my computer, both of which I hope meant I spent less time listening to music on Spotify. Did it work?

To find out, I compared my Spotify and Last.fm top artists, songs, etc. using the same time range: January 1, 2022 – October 31, 2022.

Top artists

According to Spotify, my top 5 artists of 2022 were the following:

For the same time range, my top 5 artists according to Last.fm were the following:

These are pretty similar lists! I’m somewhat surprised to see Jacques Greene in the top 5 for this time period, as well as Harry Styles. I listened to Harry Styles for a day or two on Spotify because a friend of mine loved his album Harry’s House, so I wanted to give it a fair shake. That must have been enough concentrated listening to put him into the top 5.

Top songs

According to Spotify, my top 5 songs of 2022 were the following:

For the same time range, my top 5 songs according to Last.fm were the following:

None of the same songs or even artists on both lists this time. I think this is pretty indicative of a strong discovery-to-purchase pipeline with my listening habits this year. After I realized I was listening to a song a lot on Spotify, I’d buy it (and then stop using Spotify to listen to it).

Most of the top songs from Last.fm are songs that I purchased and then listened to frequently on my phone.

Total artist and song listens

Spotify Wrapped also includes data about the total number of artists and songs listened to on their platform. Comparing that data with Last.fm leads to a couple interesting insights.

data source	total artists
Last.fm	2006
Spotify	1815

These numbers are very close together, despite the fact that I spent much less time listening on Spotify this year. It’s likely that the way that Spotify stores artist metadata for songs differently than Last.fm is coming into play again here. As I’ve noted in the past, Spotify stores a list of artists for each track as an array/multi-value field, but Last.fm assigns just one artist for a track.

It’s possible that Spotify might attempt to clean up this number to be more precise, but given that cleaning the number of artists listened to would reduce the total, I find it unlikely that the data scientists building Wrapped would make that decision. The purpose of Spotify Wrapped, after all, is marketing rather than precision.

If the data is cleaned up, the more similar totals could be explained by me doing the vast majority of my music discovery on Spotify, and use SoundCloud, YouTube, and Apple Music (iTunes) to listen to artists that I discovered already on Spotify.

That would mean that I have a large breadth of artists that I listen to on Spotify, and roughly the same number according to Last.fm (which includes Spotify listening data), again possibly reinforcing a discovery-to-purchase pipeline enabled by Spotify.

Looking at the total songs seems to lend further credence to this theory:

data source	total songs
Last.fm	4023
Spotify	2267

The total number of songs that I listened to in this time period are nearly twice as many on Last.fm compared to Spotify. This is a stark difference. It’s almost certain that my Last.fm data has some anomalies from watching YouTube videos like those from Mentour Pilot, despite my best attempts to clean that up in the data¹.

Beyond data errors, however, it’s also clear that my attempts to diversify where I listen to music are apparent in the data.

Diversifying my listens

As a grand total, Spotify calculated that I spent 12,203 minutes listening to music in Spotify between the beginning of January and the end of October. Compared with last year, that’s a dramatic drop. I listened to music using Spotify for 62,214 minutes in 2021.

If I compare the Spotify data with the rest of my data for the same time period, I get a result of 25,614 minutes listened². This means that when I was listening to music between January and October, I spent less than half of that time listening on Spotify.

What was I doing instead? Listening to purchased music in Apple Music on my laptop or my phone, listening to my SoundCloud Weekly playlists on SoundCloud, or digging into Tiny Desk Concerts, an epic live performance featuring an organ, and other music on YouTube.

I primarily listened to newer music on Spotify, using it for the Release Radar playlist and any other new albums that I wanted to check out.

Out of the total minutes for this time period, Spotify claims I spent 159 listening to Fred again… Given that I listened to Fred again.. tracks 181 times according to Last.fm, the real number was closer to 319 minutes³. I apparently couldn’t stop listening to his track Jungle, which is accurate because that was my top song for that time range.

My top song

According to Spotify, I listened to Satyr by O’Flynn and Frazer Ray the most on July 17, 2022. The track came out July 14, 2022, but I’m not sure why Spotify called that song and date out specifically.

According to my data, two days are tied for the most listens of that track:

July 17. 2022
August 16, 2022

Both days have a total of just two listens each. I think a pattern throughout my listening activity this year is that I didn’t have a lot of songs that were on constant repeat for me—and those that were, I didn’t listen to on Spotify.

Spotify called out that this song had 11 listens, and I had 13 listens recorded in my Last.fm data for the same time period as Spotify Wrapped.

Here’s the backstory of the data, at best guess:

When I realized that Satyr was a single off a forthcoming album by O’Flynn and Frazer Ray, I pre-ordered it. The album came out October 21, 2022, but Satyr was released as an album single on September 26, 2022. It’s likely that any listens of that track after that date happened in Apple Music—either on my phone or my computer.

Because I spent much less time listening on Spotify this year, the song that was truly an obsession on one day wasn’t captured in Spotify Wrapped at all. I listened to Hadiya George’s track Hot Flavor (Godmode Smash Brothers Remix) 14 times on Tuesday, August 23—more times than I listened to Satyr for the entire time period covered by Spotify Wrapped.

What Spotify Wrapped does offer, which I can’t evaluate as easily with my own listening data, is aggregated data insights.

A peek inside the Spotify data factory

As Spotify Wrapped continues to evolve, Spotify reveals more and more of their data science practices in as part of the stories about your “year” in music.

As Amanda Hoover reports in The Big Problem With Spotify Wrapped, Spotify leverages the extensive amount of data they collect—especially listening behavior of users and derived metadata about the music, like mood—to create Spotify Wrapped.

What makes Spotify so good at creating these lists and predicting the music that users want to hear is a robust artificial intelligence system and its immense data trove.

Beyond sharing personalized listening insights, Spotify has exposed their data storage methods and hinted at the mood descriptor metadata they derive from music in past versions of Spotify Wrapped.

Their data storage methods became more apparent to the average Spotify user with Spotify Wrapped 2020. In my 2020 post, I pointed out that the way that Spotify stores artist metadata affects the data they share:

Because Spotify stores all artists that contributed to a track as an array, I can listen to a track with 4 contributing artists on it, 1 of which I’ve listened to before, and according to Spotify, I’ve now discovered 3 artists and listened to 4

This was especially apparent in the genre discovery slide, where Spotify revealed the number of niche and hyper-specific genres that it assigned to artists:

According to Spotify, I listened to 801 genres this year, including 294 new ones. I’m not even sure I could name 30 genres, let alone 300 or 800. Where are these numbers coming from?

It turns out that, much like storing artist data as an array for each song, Spotify stores genre data as an array for each artist. This means that each artist can be assigned multiple genres, thus successfully inflating the number of genres that you’ve listened to in 2020.

For Spotify Wrapped 2021, Spotify took it a step further and crafted “musical auras” for users eligible for Spotify Wrapped. I explored this in my 2021 post:

My best guess is that there must be an internal API that contains mood descriptors for Spotify tracks that helps them build their emotion-based playlists. The closest metadata available in public APIs is the “energy” audio feature, which isn’t enough on its own to associate with an emotion (at least in my opinion).

This year, Spotify exposed their trove of insights with two new features:

My Spotify listening personality

I love personality tests. They give you a certain amount of insight into your behavior and personality based on questions that you answer about your activity, behavior, or feelings.

Most frameworks for deriving personality insights use questions about your behavior to draw conclusions, like attachment styles, love languages, four tendencies, Myers-Briggs type indicator, or the enneagram. That’s because the simplest way to draw conclusions from your behavior is to ask about it.

Spotify didn’t have to ask any questions to derive my listening personality—it has enough behavioral data that it collects when I use the service to listen to music.

Spotify collects user-specific behavioral data such as:

what music I listen to
which songs I add to playlists
the names of the playlists that I listen to
the time of day that I listen to music
whether I skip a track or keep listening to the end

as well as less personal data, such as:

what people are saying online about various artists
the characteristics of specific tracks, such as their energy or valence
which artists and tracks tend to get added to which playlists
the lyrics of various tracks

With all of that data about me and the music I listen to, Spotify derived a listening personality for me: The Specialist.

Similar to a Myers-Briggs personality test, Spotify gave me a personality of FNVU:

Familiarity
Newness
Variety
Uniqueness

You’re selective with the music and artists you listen to, but you’ve got lots of love to go around. Once you decide you like an artist, you’re all in.

Based on a blog post that Spotify published, Get to Know Your Music Listening Personality from 2022 Wrapped, there were 16 possible combinations made up of these 8 options (paraphrased by me from the post):

one option	complementary option
Familiarity You mostly listen to your favorite artists over and over.	Exploration You sample a lot of new artists.
Timelessness You wander the vast catalog of all the music ever made.	Newness You listen to brand new music right when it comes out.
Loyalty You find yourself going back to the same tracks and playing them on repeat.	Variety You like to spin through a lot of music before repeating.
Commonality You listen to mostly popular artists along with millions of fellow fans.	Uniqueness You look deeper for someone less well known.

Spotify, in their blog post, refers to these as four metrics. And that’s because they’re based on four metrics (although revealed to us as 8 options). The blog post doesn’t provide any additional details about these metrics, so from here on out I’m speculating.

Based on another blog post from Spotify, Everything You Need To Know About 2022 Wrapped, I’m fairly certain that this profile was calculated specifically for Spotify Wrapped over the January – October time period:

Your Wrapped will showcase your Listening Personality based on how you listened to music throughout the year.

In my opinion, these are the four metrics used to derive the Spotify listening personality:

It’s a fairly straightforward algorithmic profile of your listening behavior, and one adjacent to the one that Spotify likely already creates in order to craft recommendations and tracks for Discover Weekly, Release Radar, and other algorithmic playlists like Daily Mixes.

What do these metrics look like, and how might they be calculated?

Familiarity / Exploration

Familiarity and exploration are derived from your artist discovery listening patterns.

This metric is likely calculated as follows: given the listens during a set time period, how many were of artists discovered this year and how many were of artists discovered in previous years?

I’m able to calculate this metric with my own data ⁴.

Based on my data, this is an accurate metric for me. 84.4% of my listens were of artists discovered before 2022, while 15.6% of my listens were of artists that I discovered in 2022. Tendency toward familiar artists, confirmed.

Timelessness / Newness

Timelessness and newness are derived from the newness of the tracks that you’re listening to, based on release date.

This metric is likely calculated as follows: given the listens during a set time period, how many listens were of songs that were released in the last 18 months, and how many were of songs released before that 18 month period.

I’m assuming that with this metric, Spotify is defining newness the same way the rest of the music industry does: new tracks are those released in the last 18 months. Anything older than that is considered part of an artist’s “catalog” ⁵.

It’s even more likely that Spotify is comparing catalog and non-catalog listening behavior with this metric because that’s exactly what they did in their latest Fan Study research report, released in September 2022 based on data from April 2022.

This metric is also easy for Spotify to calculate, because they have track metadata directly from the music distributors and record labels.

I can’t calculate this metric nearly as accurately as Spotify, because I only have release date data for 12.9% of the tracks that I listened to this year ⁶. Because I get release date data from my iTunes library, I’m limited to the data that exists for tracks that I own, and only tracks that I purchased through the iTunes store contain release date metadata (I assume).

Of the 12.9% of tracks for which I have data, 18.4% of my listens in the set time period were of tracks released between April 30, 2021 and October 31, 2022. 81.6% of my listens were of tracks released before that date.

To reinforce the low data that this is based on, only 160 listens are reflected in the 18.4%, and 711 listens are reflected in the 81.6%, out of total listens of 6522 songs. It’s a tiny amount of data, but it’s different from what Spotify classified for me. I think this is largely because I use Spotify primarily to listen to new music, such as with Release Radar, before I buy it.

Loyalty / Variety

Loyalty and variety are derived from your track listening patterns.

This metric is likely calculated as follows: given the listens during a set time period, how often did you listen to the same track, compared with listening to other tracks.

I can calculate this with my own data by using a time window and looking for repeat track listens during a given time window ⁷.

95.8% of my listens are to different tracks within a given 20-minute-long time window during the set period of January 1st – October 31st. A scant 4.2% of those time windows were loyal to specific tracks. Some notable moments ⁸:

July 13, 2022 11:00AM–11:20AM, I listened to salute’s track Honey 4 times
August 23, 2022 9:20PM–9:40PM, I listened to Hadiya George’s track Hot Flavor (Godmode Smash Brothers Remix) 4 times

The tracks that I listened to on repeat most frequently (across multiple time windows) were:

I based my assumption about this metric on time windows largely due to the Spotify Research blog post, The Dynamics of Exploration on Spotify, specifically the section about The Online Lifecycle: Exploratory Phases.

In that post, Spotify researchers disclose the methods they used to identify patterns in user exploration and discovery

we looked at how much listeners explored between time windows (again, we used a week) in their individual traces and measured the discovery rate interquartile range (IQR) over windows with multiple streams.

users actually chain exploration-heavy windows together. If you already started exploring, you are more likely to keep going; if you have not started exploring yet, you are also more likely to not start in the next week. So, there seem to be phases of both content exploration and content revisitation.

It’s possible that Spotify used similar methodologies to determine loyalty or variety tendencies for the Spotify Wrapped personality metrics, but without more details, I can only speculate. For now, I can confirm with my own data that Spotify accurately categorized me as seeking out variety in my listening patterns.

Commonality / Uniqueness

Commonality and uniqueness are derived from the popularity of the tracks you listen to.

This metric is likely calculated as follows: given the listens during a set time period, how many had a high popularity value?

Spotify maintains a popularity metric for each track in their library, but I’m not sure what popularity score might be considered common compared to unique for this metric.

I don’t have a popularity metric for each song in my library, because I don’t have access to listening behavior for millions of tracks by millions of users. What I can do, is use the Spotify Web Console to call Get Playlist Items for my Top Songs 2022 playlist and gather popularity metrics for the tracks on that playlist.

artist	track_name	popularity
Harry Styles	Late Night Talking	90
Harry Styles	Music For a Sushi Restaurant	83
Beyoncé	BREAK MY SOUL	79
Harry Styles	Grapejuice	78
Burna Boy	Last Last	75
Pheelz	Finesse	74
Burna Boy	It’s Plenty	74
Fred again..	Jungle	72
Maverick Sabre	Slow Down (feat. Jorja Smith) - Vintage Culture & Slow Motion Remix	71

Based on the artists at the top end of the popularity score, let’s say that a popularity score of 75 and above makes a song common and anything lower makes it unique.

For the 100 songs that I have data for, 4.1% of my listens were to common songs, and 95.9% of my listens were to unique songs.

It’s also worth noting that I only listened to the songs on my top 100 songs playlist a total of 241 times during this time period. To get a more accurate score based on my Last.fm data I’d need a lot more data, but I think this is a pretty accurate depiction of my listening behavior overall. Tendency toward unique artists, confirmed.

Spotify Wrapped included more than just a listening personality—this year it also assigned mood descriptors to different times of day.

My Spotify Audio Day

According to the Everything You Need To Know About 2022 Wrapped blog post from Spotify:

Audio Day showcases the niche moods and aesthetic descriptors of the music you listened to during morning, midday, and evening time periods.

For me, that looked like the following

While my Audio Day descriptors were somewhat bland:

Minimalist
Chill Relaxing
Cathartic
Intense Tender
Hopeless Romantic
Chill Energy

Some of my friends had more mystifying and novel descriptors:

Acceptance Sad
Sad Boi
Zen Peaceful
Yearning
Energizing
Relaxing Calm

It’s unclear if each descriptor is a complete phrase, or if it’s actually separate descriptors, based on the seemingly random nature of the categorization.

How did Spotify define these “niche moods and aesthetic descriptors”?

Poking around on the Spotify Research blog, I discovered the post The Contribution of Lyrics and Acoustics to Collaborative Understanding of Mood, which points out:

The association between a song and a mood descriptor was calculated using collaborative data, by “wisdom of the crowd”. More specifically, these relationships were derived from Spotify playlists’ titles and descriptions, by measuring the co-occurrence of a given song in a playlist, and the target mood descriptor in its title or description.

Researchers derived mood descriptions based on the titles and descriptions for playlists that songs were added to by Spotify users. Those mood descriptors were used as the baseline for the rest of the research, which explored whether the energy of a song and the lyrical content could be reliably correlated with mood.

And indeed, the findings of the research asserted that:

by combining information extracted from the two modalities – lyrics and acoustics – we can best predict the relationship between a song and a given mood.

I think that, based on the haphazard nature of the descriptions for My Audio Day, these were likely derived from the playlist title and description data source. Spotify is performing this research, however, to improve recommendations and search. Namely:

we want to enable search based on mood descriptors in the Spotify app, for example by allowing users to search for “happy songs”. Additionally, from the recommendations side, we want to be able to recommend new songs to users that provide similar sets of moods users might already like.

If you want to find tracks representative of your Audio Day, you might want to search for the descriptors on Spotify and look for relevant playlist titles.

For my own descriptors, I found playlists like the following:

Attempting to search for “Cathartic” or “Intense Tender” or “Cathartic Intense Tender” yielded a number of user-created playlists that were exclusively Taylor Swift songs.

All of these playlists contain songs and artists that I listen to frequently, and have saved in my Spotify library. It’s also amusing to me that if you remove the genre identifier and the word “Mix” from the title, they’re near-perfect matches for many of my audio day descriptors.

Maybe my friend with Sad Boi as one of their audio day descriptors spends a lot of time listening to artists on the Sad Boi Hours playlist.

Spotify Wrapped 2022: Deriving metrics for novelty and capitalism

Why bother digging into all these data points? Spotify Wrapped is full of derived metrics that feel insightful to us, revealing our music listening patterns. For Spotify, those same insights are the backbone of the business.

Collecting data about our behavior and correlating that with computed and explicit metadata on the music and podcasts distributed through the service is how Spotify improves the listening experience on the platform. The better the experience (or the quality of the offerings), the easier it is to entice new users to sign up, subscribe to Premium, and create a captive audience for ad targeting.

This was the first Wrapped where the scope of Spotify’s correlational capabilities was made transparent to the average user. I can’t tell if people are amused, intrigued, or uninterested in these new data points.

It seems to me like the primary focus of the people sharing Spotify Wrapped is to get some new music recommendations, and be recognized for how much time you spent listening to your (possibly) favorite artists and tracks of the year.

For me, it’s always amusing to explore my data through new lenses, and occasionally frustrating to be faced with the holes or errors in my own data collection. But I’m not Spotify. I’m at the mercy of the APIs and data sources that I use to collect my music data, and the documentation that might or might not exist for those sources.

I’ll be back at the end of the month and the year with a full rundown into my top music of 2022, including details about the money earned by artists based on my music habits, what instigated various artist obsessions, and how my listening patterns were different from previous years. Thanks for reading!

Wait, what? Why would you be getting YouTube video streams recorded and messing up your data? Because there is a web scrobbler extension that I use to scrobble my listening activity in SoundCloud, YouTube, and Bandcamp. Unfortunately for my data, it’s a bit overzealous and logs videos like those from Mentour Pilot as tracks in my Last.fm history.

I cleaned these up by searching for known bad data (artist=“Mentour Pilot”) in my Last.fm data in Splunk, as well as by sorting all track names in my listening history by length and removing the anomalies. I realized that YouTube videos often have long detailed names for algorithm optimization purposes, while music tracks don’t often have a name longer than, say, 20 characters.

Using Splunk to do the math, the average track name length for all tracks that I listened to this year is 18 characters. I also looked into the longest track names that I listened to in 2022. Excluding a couple malformed data points, Lil Nas X’s track “MONTERO (Call Me By Your Name) (But Lil Nas X Makes All The Sounds With His Mouth)” is the longest at 82 characters. Most of the other long track names are remixes. The shortest track name is 1 character: Super Flu’s track 1. ↩︎

Using a rough calculation that uses the track length from iTunes, if I had the track in my library, or 4 minutes if I didn’t, as evidenced in this Splunk search using an eval coalesce function:

`lastfm` |  eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false")
| search exclude=false
| fields track_name,artist
| lookup itunestracklength.csv track_name OUTPUT track_length
| eval length=coalesce(track_length,240000)
| stats count, values(length) as duration by track_name,artist
| eval total_time=(duration*count) | eval mins=round('total_time'/60000,2)
|  stats sum(total_time) as real_total
|  eval total_artist_time=round(real_total/60000)

Four minutes is represented as milliseconds, to match the duration that I derived from my iTunes data and simplify my math. ↩︎

According to the following calculation, which uses the metadata from my iTunes library to supply the track length for the Fred again.. songs, because I own his discography:

`lastfm`
| eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false")
| search exclude=false
| fields track_name,artist
| lookup itunestracklength.csv track_name OUTPUT track_length,ui_track_length
| search track_length=* artist="Fred again.."
| stats count, values(track_length) as duration by track_name
| eval total_time=(duration*count)
| stats sum(total_time) as real_total
| eval total_fred_time=round(real_total/60000)

This isn’t necessarily the most efficient way to do the search, but it works. It also excludes a morning of bad data from the results. ↩︎

I keep a lookup that identifies the first time a track appeared in my Last.fm listening history, which has existed since 2007. Based on that data, I can then calculate when I first discovered a track. If I use that data then I can see which songs I discovered in 2022 and also listened to in 2022, compared to listening to tracks from before then.
```
index=music sourcetype="lastfm*"
| dedup uts
| lookup earliestlistens.csv artist OUTPUT year as earliestyear
| eval thisyear=if(earliestyear==2022, "exploration", "familiar")
| stats count by thisyear
```
It’s a pretty simple but helpful way to identify music discovery patterns. ↩︎
If you want to read further about the music industry definition of catalog, see Does the music industry’s definition of ‘catalogue’ need an upgrade? on Music Business Worldwide. ↩︎
I can correlate my Last.fm listening activity for the given time period with the release date information from my iTunes library, which I pull out in an automated fashion into a lookup to make it easier to correlate with event data. I used the epoch timestamp for April 30, 2021 as the stand-in for the catalog cutoff, 18 months before October 31, 2022.
```
`lastfm`
| fields track_name,artist
| lookup track_release.csv track_name, artist OUTPUT release_date
| search release_date=*
| eval newness=if(release_date>1619784000, "new", "old")
| stats count by newness
```
I considered picking a random release date to assign to all the unknown tracks, but imputing that data would be extremely difficult to do in a reliable or accurate sense. If you have any ideas, please let me know! ↩︎
I do this by separating all of my listening activity into 20-minute-long bins, then counting the track names in each bin. For any bin where I listened to the same track more than once, I deemed that “loyalty” to a track, otherwise my listening activity was deemed variety-seeking.
```
`lastfm` |  eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false")
    | search exclude=false
    | bin _time span=20m
| stats count by track_name,_time
| eval habit=if(count>1, "loyal", "variety")
| stats count by habit
```
After I determined whether I was loyal to a specific track during a given time window, I counted each instance of loyalty or variety (one for each track_name within a given time window) and graphed it. ↩︎
Expanding on the previous search, I looked for only the tracks that I was loyal to, and sorted by count and time window:
```
`lasfm`
| eval exclude=if(uts>=1654688186 AND uts<=1654723907, "true", "false") | search exclude=false
| bin _time span=20m
| stats count by track_name,_time
| eval habit=if(count>1, "loyal", "variety")
| search habit=loyal
| sort -count, _time
```
Unfortunately, this mostly revealed a lot of data errors with the way the Last.fm mobile app records and submits data. A lot of my “most loyal” time windows were a result of double or triple scrobble submissions by the mobile app. I’m still digging into the cause of that but I think it’s related to how Apple Music stores the “listening history” on the device. ↩︎

Where to start with analytics for documentation

Sun, 27 Nov 2022 10:58:56 +0000

It’s tough to find helpful information about analyzing website metrics for technical documentation sites.

The goals of technical documentation are different from those of a marketing blog or a company website. You’re not optimizing for maximum traffic. No one is clicking “Add to Cart” on your API reference topics.

You need to use slightly different metrics and in different ways than you might for a marketing blog or your company’s website.

If you want to start using site analytics with your documentation but aren’t sure where to start, this post is for you.

What are site analytics?

Website analytics, in the context of this post, are the metrics collected about a user’s activity when they access a website—in this context, the website hosting your product documentation.

Those user metrics are then aggregated and made available as website analytics such as page views, unique users, average session length, page referrer, and more.

Typically, companies add analytics when they launch their websites, so your documentation site almost certainly has a tool like Google Analytics, Adobe Analytics, or a smaller provider like Hotjar set up.

Get a basic understanding of working with site analytics

Start with these blog posts from Bob Watson:

Those blog posts are an excellent foundation for building a basic understanding of what site analytics mean.

Make sure to get familiar with the tool that you’re using to analyze the data¹. If you’re focusing on one section of your documentation, learn how to filter the data so that you only see relevant values².

You also want to look at site analytics with several months worth of data. For example, as Kumar Dhanagopal points out in his talk for Write the Docs Portland 2022, Don’t trust the numbers!, it’s more valuable to look at trends rather than absolute numbers.

With that in mind, don’t look at analytics for new pages until a few months have passed, to make sure the metrics are relevant and reflect a consistent pattern.

Start looking at your site analytics

As tech writers, we’re frequently understaffed and overburdened. There’s always more writing to be done than we can tackle. Because of that, we need to be strategic.

Data can help when we’re trying to figure out how to do more work with our time.

Maybe these scenarios sound familiar:

I have a lot of outdated content that I need to update, but where do I start?
People keep asking questions in Slack and in forums that are answered in the documentation, but why can’t they find what they’re looking for?
My boss wants to put all the documentation inside the product, but I’m not so sure.

Site analytics can help with these scenarios if you can ask specific, discrete, and data-focused questions of the data. In this way, website analytics function as imperfect proxies³ for the information that you actually want to know.

Define and refine your questions

When you initially come up with a question, it might not be something you can answer with data. What you can do is refine what you want to know into a more data-focused question, and then identify which website analytics metrics might help you answer it.

What you want to know	Data-focused question	Website analytics metric
What do people find useful?	Which pages are most popular?	Page views
Where should I start updating outdated content?	Which pages are most viewed?	Page views
Is the documentation easy to find?	How are people getting to the documentation?	Channel, referrer, and source
What do people want to know about the product?	What searches are leading people to my documentation?	Search terms
What portion of our overall user base uses the documentation?	How many users are viewing these pages?	Total users
Are people finding the information they need? Are they lost?	Are people clicking links and engaging with the documentation?	Click-through and engagement data

Although these are data-focused questions, data is rarely, if ever, definitive. You can’t expect an obvious answer to any questions you ask of the data.

Instead, your goal is to reduce your uncertainty about the answer. When you ask a question about your documentation, such as “How are people getting to the documentation?”, you can make assumptions about the answers.

Maybe people are searching, using a browser bookmark, or opened a link from an email or chat message.

After you make assumptions, you can look at the data to reduce your uncertainty and validate or reject your assumptions.

Which pages are most popular or most viewed?

If you want to know which pages are most popular or most viewed in your documentation, look at page views. Page views are exactly that — when someone views your page in a web browser.

How to look at page views

Look at page views across different time scales, to identify which pages are consistently viewed, and which ones are frequently viewed.

If you look at the page views for a month, look at the previous month, and the same month last year, to help you get a sense of baseline page views.

As Bob Watson points out in How you can make sense of your site analytics, it’s important to understand the baseline number of page views for your documentation site, and use that baseline to evaluate outliers for the pages that you want to learn more about.

I prefer to use a bar chart or a table to look at page view data so that I can actually figure out what I’m looking at. Bar charts help me identify outliers at a glance, and tables give me easier to read data.

Ultimately, you’re checking the data against your expectations and a baseline.

Evaluate against your expectations

You might see page views for a certain page spike during a week. This is when you start to add context and consider the possible causes of that spike:

You could look at page referrers to see if your page is being shared somewhere like Hacker News or Reddit and going viral for some reason.
You could also consider the product marketing and release schedule—if it’s a page related to a newly announced feature, it makes sense that the traffic would be higher than usual!

Beyond patterns in page views, I’d expect that the homepage for the documentation and a “quickstart” or “installation” topic are going to be the most-viewed and most-consistently viewed pages of your documentation. This might be true for you as well, but this is something else you can explore when you contextualize the data.

How are people getting to the documentation?

If you want to know how people are getting to the documentation, there are several metrics that give you insight into this at different levels of granularity.

If you want to know which social sites, search sites, or other websites users are using to get to your documentation site, session source has that information.

The page referrer helps you find out what path to and through the documentation users are taking, if any.

What metrics to look at

Channel data gives you information about the method that someone used to get to your documentation site. The following channels⁴ are common for documentation sites:

Direct
Organic search
Referral

Session source is the disaggregated form of the default channel grouping. Looking at the session source combined with the medium lets you see both the channel and the specific site. For example:

session source	medium
search	google
search	bing
referral	t.co

Use the page referrer metric if you want to know exactly which page sent users to your documentation. If you’re using Google Analytics 4, you’ll need to do some configuration⁵ to use it in reports.

Look for surprises

When you’re looking at referrer data (whether it’s channel, session source, or page referrer), keep an eye out for surprising or unexpected sources of traffic.

You might discover that a company confluence page shows up as a frequent page referrer to your documentation. As part of contextualizing the data, you can find out whether that company is a customer of your product.

If they’re a customer, you might want to find a way to reach out to them for a user research interview.
If they’re not a customer, you might want to dig into what their company does to learn more about why they might be linking so frequently to your documentation.

What searches are leading people to the documentation?

If you want to know which search terms lead people to the documentation, use search term data. It might seem obvious that search terms are a valuable source of information, but there is some nuance in terms of which search terms are available to you.

What search term data to look at

The search terms collected by Google Analytics and similar tools are the terms used to search your documentation site while on your organization’s website. Search terms in Google Analytics are not search terms that users are typing into Google⁶.

Instead, to see search terms that users are typing into Google that might lead to your website, you need to use the Google Search Console. You can see up to 1000 search terms for which your pages made an appearance in Google Search results. If you spend some time and effort customizing Google Search Console, you can see more helpful data⁷.

What Google Search Console can’t offer you, besides the search terms that people use on other sites like Bing, Duck Duck Go, Baidu, or Yandex, is search terms in context with a user journey.

The search terms available in your analytics tool can be more useful because you can identify which terms led people to specific pages.

What to consider when using search term data

Search terms are often more valuable for large documentation sets or products with a large user base, for the following reasons:

The larger a documentation set, the harder it is to find information by browsing a table of contents, so more people will use search.
The larger the user base, the more people will use search because you have more people using your website in general.

Because search terms are a proxy for what your users want to know more about, you can draw more relevant conclusions (or more confidently validate assumptions) when you have a high volume of search terms.

If your documentation set is about 10-30 pages, or you have a small user base for your product, you might not have many search terms, or enough search term traffic, to give you helpful information in search term data.

Evaluate your search results data

When you evaluate your search results data, whether in analytics or in Google Search Console, you want to look for the following:

What are the most consistently searched terms over time?
What are the most frequently searched terms?
What are the least-searched terms?
What are the inconsistently searched terms over time?

If you have enough search term data, you can start to contextualize the data with other types of information.

You can identify successful searches, where a search term led users to a relevant documentation page, and unsuccessful searches, where a search term led to no engagement or led users to an irrelevant documentation page.

If you have a high volume of unsuccessful searches for a specific term or related group of terms, that might indicate an opportunity to write new documentation or recontextualize existing documentation to use similar terms that people are using to search.

A lack of terms in search that you might expect to see could indicate that users are finding content a different way — by navigating directly to it using bookmarks, or using the on-page navigation and landing pages rather than using search.

How many users are viewing these pages?

To find out how many users are viewing the documentation, review the total and active users⁸ for your site. I’m intentionally referring to users viewing rather than reading the documentation, because we can’t know what anyone is doing when they’re on your page⁹.

While real time users is a metric calculated by Google Analytics, it’s not valuable as a documentation metric. Your documentation site is not an ecommerce platform attempting to convert viewers into purchasers or a SaaS app attempting to measure user activity for monetization purposes.

A high number of active users might indicate useful documentation because people are returning to it consistently. However, you can’t know why users are returning to the documentation without talking to them.

I find it more valuable to focus on total users and active users to get a sense for how many of your product customers are using the documentation when you contextualize the data.

Are people clicking links and engaging with the documentation?

If you want to better understand whether people are clicking links and engaging with the documentation, you can look at click-through and engagement data.

What engagement metrics to look at

Identify how people are engaging with your documentation with the following engagement metrics:

Link URL tells you which links people clicked in your documentation, and how many times.
Bounce rate tells you the percentage of sessions in which a user visited a page in your documentation and left that page within 10 seconds without doing anything.
Average engagement time tells you the average amount of time, for a given time period, that someone viewed your documentation.

These definitions and names can vary by analytics tool, but the meanings are roughly the same.

Evaluate your engagement metrics

To evaluate these metrics, look for outliers in activity and patterns, and consider your assumptions about user behavior.

When you look for outliers, you might look for the following:

High numbers of clicks on specific links compare to other links.
High time on page or engagement time compared to other pages.
A high bounce rate for a page.

You also want to contextualize these outliers in terms of frequency and volume. If one or two people are bouncing off a page out of a total user cohort of 10K, that’s likely meaningless. If thousands of people are bouncing off one page in the same total user population, there’s probably something worth investigating.

Recognizing patterns in the data that can indicate how people are behaving can also be useful. For example, if you created a supertask topic that contains links to 6 steps in a complicated decision tree process, you could gather the clickthrough data for that topic and see which links are being clicked most frequently, and from which page.

Relatedly, if you have a wayfinding navigation topic, you could gather the clickthrough data for that topic and determine if people are using the links in that topic at all, or not. If not, they might be using search or the table of contents instead of the links in that topic. You need to add context to reliably validate your interpretations.

Maybe you have a page in your documentation that introduces the API for your product, then links to an auto-generated API reference site. You can check your metrics to determine whether, as you might expect, that link gets a lot of clicks.

For most engagement data, you want to contextualize the data. You need to know information about the documentation to make assumptions about whether a low engagement rate on a page is meaningful. If it’s a landing page, maybe you want a low average engagement time, while you might expect a higher average engagement time on a reference page with a lot of function descriptions.

Engagement metrics are ultimately limited in their usefulness. They can’t communicate people’s motivations, so you can’t draw reliable conclusions about why people are clicking links, what’s contributing to a bounce rate, or provide a reason for a high or low engagement time. Engagement isn’t a straightforward proxy for reading.

If you really want to know more about people’s behavior and motivations, supplement these metrics with added context and user interviews.

Contextualize the data

When you interpret your analytics to try to validate your assumptions about the answers to specific questions, you need a thorough understanding of the data values and the likely causes.

It’s crucial to add context to your documentation site analytics to help identify the causes of unexpected data points, such as outliers.

With context, you can try to understand why your page views spiked on December 15th, or the email channel became a dominant referrer for a week in June.

When it comes to documentation site analytics, I find it most helpful to add the following types of context:

Add behavior context

Talk to users to add context to user behavior.

Beyond how many people are viewing the documentation, you also want to know why they are viewing the documentation.

Talking to users is the only way to answer questions like “why are people staying on the SDK function reference page so long?” or “how do you typically get to the documentation and why?”.

I always recommend supplementing site analytics with user interviews, as Bob Watson also recommends on his blog post, Tips for conducting documentation research on the cheap.

If you have access to a user experience researcher, work with them to conduct some high quality user research about your documentation. The Write the Docs website has helpful resources about user research.

Ultimately, the best way to know if your documentation is useful and valuable is to ask the people you’re writing it for.

Add product context

Add product telemetry and information to add context to your website metrics. It’s difficult to evaluate a baseline or understand patterns in your documentation metrics without an understanding of user behavior in the product.

You can do this in a number of ways.

Correlate with product telemetry

If you can, I recommend correlating site analytics with product telemetry data. Very few folks have an end-to-end data surveillance apparatus to follow users from web application to documentation site with one session cookie.

Instead of a complex approach, stick to the basics. Compare rough numbers of site users to rough numbers of product users. It’s only helpful to do this at a large enough scale to where rounding and measurement errors won’t affect any conclusions that you draw.

This context can help you evaluate whether a higher-than-average number of views for a documentation topic about a particular product feature is happening at the same time that the product feature itself is seeing a higher-than-average number of active users.

Beyond active users, you can talk to product managers and other internal stakeholders, to get an understanding of what product-related metrics and information they use. Some of it might be helpful to you.

Consider the product functionality

In addition to product telemetry, consider what you know about the product itself.

You’re likely to see more page views (and feedback) for topics about popular existing product functionality.

Maybe your product uses in-app documentation for some functionality, such as auto-complete for commands. If you see lower-than-expected page views for those pages, the in-app documentation could explain why.

As another example you might see frequent searches for a specific phrase that isn’t in the documentation, but in reviewing the product itself, discover that the product uses that phrase in the user interface. That’s valuable context to explain why customers are using that search term to look for documentation.

Add documentation context

Perhaps the easiest context for tech writers to add when reviewing site analytics is context about the documentation itself.

Consider topic type

You always want to consider the type of content you’re evaluating:

Is it task content?
Reference content?
Concept content?
Intro content?
A combination of all of the above?

Topic type matters because users behave differently on different types of content pages. For example, readers might look at a reference topic briefly after locating the information they need, but spend awhile on a concept topic reading and making sense of the information.

If your documentation isn’t strongly typed, you might not notice differences in clickthrough rates or time on page that correlates with topic type.

Consider content goals

You also want to consider the goals of that content, or the purpose of the page, to inform your expectations about how a user might interact with it.

If you’re writing in-depth content like a tutorial or an interactive scenario, you might expect someone to stay on those pages for a long period of time and click links only to other parts of the tutorial.

topic path	average engagement time	page_views	link URL	click_events
/docs/user-getting-started	02:01	1436	https://example.com/docs/user-next-steps.html	1201

On the other hand, if you’re writing reference content like a command or function reference, you might expect someone to quickly bounce off the page after finding the name of the command they couldn’t remember. Alternately, the page might have a very long time on page from a developer leaving the SDK reference page open while they write a script.

topic path	average engagement time	page_views
/docs/command-ref-init	00:20	554
/docs/command-ref-all-transforms	00:55	1203
/docs/sdk-function-ref	3:07	387

For a page that you wrote as a navigation or landing page to introduce functionality, you might want to check to see if that page is a common entry point for the documentation, and whether people are clicking the links on that page to get to subtopics.

topic path	page_views	link URL	click_events
/docs/get-started	3409	https://example.com/download	1807
/docs/get-started	3409	https://example.com/docs/user-getting-started	1436
/docs/get-started	3409	https://example.com/docs/install	1208
/docs/get-started	3409	https://example.com/docs/develop-start	604

Many users find information by searching, but there are still users who browse the navigation of websites to find what they’re looking for¹⁰. In that case, if more people are landing on subsections instead of the navigation page, you might conclude that more searchers than browsers are using your website.

Consider content scope

Beyond content type and goals, you also want to consider the scope of the content on a page.

A page that documents functionality that’s only relevant or visible to a small portion of customers is a page that you would expect to have a lower number of page views, or perhaps low engagement.

The methods that people use to get to a page might vary for alpha, beta, or preview content compared to content about a generally available feature. You might expect to see more direct links for content that documents early-stage features, as sales people share links directly with customers. Content about a generally available feature is likely found most frequently from search.

Consider content age

New content, or newly updated content might have a different page view profile than other pages. It might not get much attention because it isn’t indexed fully by search engines yet, or it might get a lot of attention depending on the business and industry context of the feature covered by the page.

When you publish a new documentation page or a new content rearchitecture, avoid looking at the metrics immediately after you publish. Leave time for the page views to settle into a consistent pattern before drawing in-depth conclusions from the data.

Older content that hasn’t been updated in some time might not get a lot of page views. It might get a lot because maybe it’s the canonical topic for installing your product, and the installation process hasn’t changed in months. Content age alone isn’t enough of a signal.

Add business and industry context

You also need to add business and industry context to your analytics to make sense of patterns and outliers that you identify.

Consider what your company is doing, and how that might affect your documentation site analytics:

Company behavior	Possible analytics pattern
Marketing department starts a new campaign with specific terminology	Increase in search terms with the same terminology
Company hosts a three-day conference	Higher page views before, during, and after the conference for new feature pages. Lower than average engagement during the conference
Leadership interviewed on a popular podcast or television show	Higher page views on the documentation homepage. Novel search terms or higher volume for basic search terms.

You also want to consider what is happening in the industry around your company or organization:

If your product was included favorably in an analyst report, such as the Gartner magic quadrant, page views and searches for your documentation might increase.
If your organization has been featured on a popular website or email newsletter, you might see a new channel or session source on your documentation site.

You can add helpful context to the metrics you’re trying to analyze with a good understanding of the business and industry context for your product and documentation.

Conclusion

It’s tough to know where to start using website analytics with documentation, especially when the tools and resources primarily address different use cases.

Analytics are intimidating, but remember, they’re fuzzy too. Fuzzy and imperfect proxies for what we really want to know: Is our documentation helping?

Despite being imperfect proxies, it’s useful to look at site analytics to reduce our uncertainty about the answers to specific, discrete, and data-focused questions.

After reading this post, I hope you can more confidently look at site analytics data, evaluate it against your assumptions, and add valuable context about user behavior, product usage, documentation types and goals, as well as business and industry activity.

What next?

Watch Kumar Dhanagopal’s presentation at Write the Docs Portland 2022, Don’t trust the numbers!
Subscribe to Bob Watson’s blog, Docs by Design.
Watch my talk or read my blog about adding data to your documentation prioritization process.
Level up your data analysis by exporting your data to a more robust tool where you can do advanced analysis and visualization.
Dive into the resources on a site like Analytics Demystified to get more familiar with the analytics platform that you’re using.
Explore other data sources, such as metadata about your documentation like topic length, header length, readability, and more. Kumar Dhanagopal covers this in his talk as well.

If you’re using Google Analytics 4, get familiar with what you can do in Reports and Explorations. ↩︎
For example, say you’re using Google Analytics 4 and your documentation is all on one domain. You can filter on hostname and focus on only that content. If your content is only relevant for a specific path, well, now you’re struggling to build a filter in GA4 because it looks for exact regex matches by default, whereas UA did partial matches and also supported in-table regex filtering. If you need to match a variety of content such as /docs/guide/admin-intro.html, /docs/guide/configuration-guidance.html, /docs/guide/config-details.html, /docs/guide/administrator-reference.html and others, I recommend using Explorations and creating a custom Events Segment with regex matches for the paths. In this case, one condition would be Page path and screen + matches regex + /docs/guide/admin.* OR Page path and screen + matches regex + docs/guide/config.* to match all the page paths starting with admin or config and map to the relevant topics. Create as many filter conditions as you need, and maybe wish that past you (or another past writer) was a bit more consistent with topic title naming in the URL path! ↩︎
As Kumar Dhanagopal points out in his talk at Write the Docs Portland 2022, Don’t trust the numbers!, Offline and secondhand usage is missing from the data. Anyone who blocks cookies isn’t tracked, and people who download PDFs of the documentation aren’t recorded either. Remember that website analytics data do not provide an exhaustive accounting of people viewing your documentation. ↩︎
The Google Analytics 4 documentation on Default channel grouping provides the rules that comprise these definitions, but the descriptions aren’t that helpful. Thankfully, a blog on How to Change Default Channel Settings in Google Analytics has useful descriptions. ↩︎
Page referrer is a parameter and not a top level dimension that you can add to reports in Google Analytics 4. As a result, you’ll want to set it up as a custom dimension so that you can add it to tables. See the Google Analytics 4 documentation on Custom dimensions and metrics. ↩︎
Google encrypts user searches, and as a result, doesn’t share individual search terms that might lead users to your site with any analytics tools including its own. ↩︎
This blog post on Search Engine Land about How to get the most out of the Google Search Console API using regex provides useful guidance for making sense of Google Search Console. ↩︎
For a definition of total users and active users, see the Google Analytics 4 documentation on Analytics dimensions and metrics ↩︎
Google Analytics 4 includes a scroll event with a percent_scrolled parameter, but knowing how far someone scrolled on your page doesn’t tell you if they were reading your documentation. Because documentation is usually scanned or referred to rather than read, identifying the scroll percentage for a documentation topic is not helpful. ↩︎
Tucker FitzGerald addresses these two types of patterns in depth in an article for UX Booth, Searchers and Browsers: the Personality Types of UX. The excellent entry on How people find information in the Australian Government Style Guide pointed me toward his article. ↩︎

Will we see prompt-based music generation?

Wed, 02 Nov 2022 08:10:55 +0000

Image generation tools like DALL-E 2, Midjourney or interfaces built on top of Stable Diffusion have gained a lot of popularity, both due to the quality of images created and the novelty of transforming text input into a visual output.

The other aspect of their popularity is that these tools let people without art or design training use text to describe what they want to see, and then (hopefully) get some images that resemble what they imagined.

It seems like an effortless way to avoid paying artists for stock photography, custom illustrations, and other work. Already, Shutterstock has integrated DALL-E 2 into its service, and so has Microsoft.

Artists are now in competition with prompt-based image generators. Will musicians soon be competing with prompt-based music generators? Is it worth the effort to try to replace musicians?¹

Why generate music?

Music is everywhere, even where you don’t actively think about it.

Short clips of music appear in news broadcasts between segments, as podcast intros, outros, and background music.

Instagram reels and TikToks also feature short clips of music, as do transitions between scenes in a sitcom.

Longer clips of music show up as the background music for an explanatory YouTube video, or even as a television or film soundtrack. Most restaurants, office buildings, and even shopping malls play music continuously. My office even plays music in the bathrooms.

Technically, all of that music needs to be licensed. Licensing music costs money.

If you could generate music using machine learning, you could create royalty-free music for any or all of these use cases. No licenses, no payments. A prompt-based music generator could make it easy for anyone to create that music.

Is anyone building a prompt-based music generator? Does one already exist? It’s complicated.

A brief summary of existing music generation tools

Music generation is far from a new idea, even using machine learning. However, most music generation tools use audio as the input, or prompt.

Audio clips as prompts

In April 2020, OpenAI released Jukebox, which:

“generates music, including rudimentary singing, as raw audio in a variety of genres and artist styles.”

However, Jukebox doesn’t use a prompt-based interface.

The same organization that created Stable Diffusion, Stability AI, is working on audio generation next with HarmonAI. So far, they have a tool called Dance Diffusion which requires you to train a model on music that you want to generate more of. A contributor named Markov Pavlov created a guide on How To Train Dance Diffusion AI To Generate Sounds.

HarmonAI has a Google Colab notebook linked on their GitHub profile that includes guidance for generating random samples, regenerating your own sounds, and interpolating between two sounds. HarmonAI seems to be taking a similar approach to Stable Diffusion with Dance Diffusion by releasing the model framework and leaving any sort of prompting functionality or user interface to be built by others.

Spotify released a tool in June 2022 called Basic Pitch which takes audio input and outputs the same audio transcribed in MIDI format. This isn’t a music generation tool itself, but the author of the blog post introducing Basic Pitch works on music creation research at Spotify, indicating that this work could be a stepping stone for actual music generation based on audio input.

As for other organizations in this space, in early September 2022, GoogleML released a tool called AudioLM, modeled after large language models like GPT-3. AudioLM works by supplying a few seconds of audio as a prompt, then the model continues the audio based on the prompt.

This is one step closer to a prompt-based music generator, in that it uses a prompt to generate responses, but in using audio to generate audio, it lacks the cross-media generation in use by image generation models like DALL-E 2 and Stable Diffusion, which use text prompts to produce images.

Beyond releasing AudioLM, Google Research also supports the Magenta Team, which “is an open source research project exploring the role of machine learning as a tool in the creative process.”

Song characteristics as prompts

Using machine learning as a tool for music creation or music production is popular. Many new businesses are gaining traction by offering various machine-learning-based generation services, such as Soundful, profiled in an interview on Music Business Worldwide; SoundDraw, Starmony; BandLab’s SongStarter; Splice’s CoSo and others.

All of those businesses share a common prompt approach, in that they let users of the service specify song characteristics like genre, mood, tempo, and more to prompt the music generator.

Other companies that work on creating generated music, such as Jukedeck, have already been acquired. Jukedeck was acquired by TikTok parent company ByteDance, and I couldn’t find any information about how it works.

Beyond these recent companies and tools, there have been looping tools and samples available to assist music production as long as synthesizers and music production software have been around.

Ryan Broderick’s experience using a tool called MACE, which “utilizes the power of machine learning and artificial intelligence to generate new samples as defined by categorized model’s [sic].”, led him to comment in a recent issue of his email newsletter Garbage Day:

I think we’re getting closer every day to a fully-functional DALL-E-like tool that can handle at least parts of the music production or songwriting process.

There are a lot of options to generate music using machine learning and artificial intelligence, and this overview barely covers what’s available. However, none of these tools support text prompts to generate audio and music. Why is that?

Probably because it’s hard enough to generate music at all.

Music generation is hard

Music is more than just sound. Music has component parts, like lyrics, a melody, a harmony, a bassline, a drum track, and all of those need to be pitched in the same key, at the same tempo—and sometimes change key and/or tempo within the song.

While it’s possible to write an algorithm to generate a specific type of music, if you want to generate any type of music or audio based on an arbitrary input, that is where you need machine learning—more than just one algorithm.

An article from 2020, A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions lays out 3 levels of music generation:

score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directly.

Let’s break down that third level of music generation, audio generation, into more granular levels. I think the levels for generating good music are something like the following:

Generate a single instrument melody.
Generate a dual instrument melody and harmony.
Generate 15–30 seconds of multiple instruments, with a melody, harmony, and rhythm.
Generate 2 minutes of multiple instruments with a melody, harmony, rhythm, and an actual song structure.
Generate the same with vocals.
Generate the same with vocals singing actual words.

I think the most difficult challenges in making it through all those levels are the following:

generating a melody, harmony, and rhythm that complemented each other.
generating that, but in a song structure like “verse, chorus, verse, bridge, chorus, verse”.
generating that, while adding coherent vocals and lyrics.

When it comes to the machine learning techniques that you need to tackle those levels, OpenAI’s blog post introducing Jukebox goes into some detail in the Motivation and Prior Work section.

To paraphrase that section of the blog post, there are a number of ways to generate music and those depend on how you computationally represent the music:

Symbolic generators require you to “specif[y] the timing, pitch, velocity, and instrument of each note to be played”, representing the music as those specific elements.
Raw audio encoding requires breaking music down into individual “timesteps” to represent each different type of audio in a track. As OpenAI points out, “A typical 4-minute song at CD quality (44 kHz, 16-bit) has over 10 million timesteps” due to the complexity and length of music audio.
You could also combine those two approaches.

Beyond those options outlined by OpenAI, you could also use a diffusion model, as described in the NVIDIA blog post, Improving Diffusion Models as an Alternative To GANs, Part 1².

Even after you decide which approach to take to generate music, then there are the limitations of the generated audio to contend with. OpenAI’s blog post about Jukebox covers the limitations they encountered:

while the generated songs show local musical coherence, follow traditional chord patterns, and can even feature impressive solos, we do not hear familiar larger musical structures such as choruses that repeat.

Ultimately, these challenges and limitations exist because while machine learning (ML) researchers talk about “teaching” models, and developing models capable of “self-supervised learning”, it’s tough to figure out what the models are actually “learning”.

Are generative and diffusion models capable of emulating song structure? Song structure doesn’t seem to be explicitly encoded in common audio feature datasets, but there is likely enough information in the datasets to interpret it, with a change in key, tempo, energy, or other feature combinations. It’s another challenge for generating music.

Music generation is hard. Generating music based on an arbitrary text input is harder. But let’s dig into the imaginary nuts and bolts. What would you need to make a prompt-based music generator?

What’s needed to build a prompt-based music generator?

First, you’d need a lot of high-quality data. Specifically, you’d need the following:

A large database of music that encompasses a wide range of genres and styles.
A large amount of metadata about that music, describing the characteristics of the music in quantitative terms that can be algorithmically modeled.
A large amount of text that describes characteristics of the music, that can be mapped to the quantitative characteristics.

You need the music itself so that you can identify the relevant metadata needed to train a machine learning model, or even to train a model by playing the music itself.

You need the metadata because you need to provide a quantitative representation of the audio and musical characteristics of a track. If the characteristics are stored as quantitative metrics, they can be manipulated by algorithms and used to train a generative machine learning model, or network of models.

You need the text in a metadata database so that you know which terms might be used to describe music, and the associations between those terms and the metrics that quantify the instantiations of those terms in the music.

Beyond the data, you’d also need the expertise and the computing resources to develop and run generative networks that could produce music—plus the budget to pay for the people and compute. What might it take to build one?

What if we built a prompt-based music generator?

From a usability perspective, it’s pretty simple to imagine a user interface for a prompt-based music generation tool.

You write a prompt, and the service returns an audio clip created according to your prompt.

You could also add some interactive options to more granularly control options such as the length of the output, the tempo of the output, and whether or not it includes vocals.

For example, if you prompt the service with “UK garage track with trumpets” and select a BPM of 130, you might get something like the following:

It’s all fine and good to design the user interface, but what about the backend infrastructure that’s actually going to do the music generation? From a technical perspective, this is where it is toughest.

There are a lot of steps involved in designing the backend infrastructure that could support generating music from a prompt like in this web interface mockup.

From my armchair machine learning understanding, here’s a high level of what those steps might involve:

Collect or create the metadatasets.
Clean and validate the metadatasets, or hope scale makes up for lower-quality metadata.
Write algorithms to process the metadatasets and infer relationships between the text and quantitative metrics.
Train generative adversarial networks, or a different type of generative machine learning framework, on the relationships between the metadatasets and on the audio correlated with the metadata.
Review the output and make adjustments to the model training and data accordingly.

There’s a lot of complexity wrapped up in those steps, as well as a lot of time and money. Who could tackle this problem effectively?

Who could build a prompt-based music generator?

Only a few companies have the data and machine learning capabilities to build a tool like this:

Spotify
Apple
Pandora
Gracenote (Nielsen)

All of these companies are uniquely positioned—they have explicit rights to access large swaths of recorded music, and the business cases to create detailed and accurate metadatasets for recommendation engines or for sale, as is the case with Gracenote.

Spotify has a music creation research area described as follows:

Spotify is developing novel technologies for AI-assisted music creation. We contribute to research in music modelling and music generation, both in the audio and symbolic domains, as well as developing tools for creators that use Spotify data (catalogue, stems, skip profiles) with the goal of helping artists create new musical material, styles or trends. In addition, we are defining new music creation/composition strategies adapted to the use of AI in collaboration with artists. Techniques developed range from sequence modeling (deep networks, belief propagation), audio concatenative synthesis, deep learning audio generation and many others.

But they haven’t released any music generation tools yet.

Apple, with the acquisition of Shazam, has even more layers of music metadata with the audio fingerprints stored by Shazam as well.

Google, despite having YouTube Music, only has explicit metadata about the track artist and other information and doesn’t have the extra high quality metadata layers specific to music. As such, Google would likely not be able to build something like this. However, Google does have a machine learning research organization.

Industry journal Music Business Worldwide points out that actually, the leader of AI-based music generation could be TikTok’s parent company, Bytedance. In TikTok goes on AI music-making and machine learning hiring spree, Murray Stassen covers hiring and acquisition trends by the company, making the case that they’re making focused advancements in AI-based music generation.

Outside of these companies, or the ability to license datasets from one of them, you’d have to rely on datasets like this free list of audio metadatasets, and/or the Million Song Dataset.

If one of these companies decided to build a prompt-based music generator, what challenges might they face?

Challenges with building a prompt-based music generator

There are lots of challenges with building a prompt-based music generator. We’ve already established that it’s a technical machine learning challenge to generate music, but solving those technical challenges aren’t the only hurdles for a prompt-based music generator.

I think the biggest challenges can be organized in the following categories:

Tools exist today to generate music based on audio, just without a text-based prompt interface, so it’s possible that some of these challenges have already been solved.

Let’s start with the usability challenges specific to a text-based prompt interface.

Usability challenges

There are some advantages to using text to prompt music generation instead of sound. If you’re tone deaf or have no sense of rhythm, you could still create some music.

It also avoids the challenge of having to record an audio file and share it with a website, but that’s a pretty trivial challenge if you’re already using the high amount of computational power necessary to locally generate, or internet bandwidth necessary to generate in the cloud and retrieve your creation.

Describing music with words is hard

The most glaring usability challenge for a text-based prompt interface is that it’s hard to describe sound with words.

If you’ve ever tried to explain how a song sounds to someone who has never heard a song by the artist before, you realize how much ambiguous referential terminology you end up using.

You pull out phrases like “it’s like house music, but much darker” or “it has the four-on-the-floor beat but is kinda ravey at the same time” or “it’s like if Led Zeppelin did acid, but faster” until you finally succumb, pull out your phone, and play the track.

Another challenge is the type of terminology to support. As a relatively untrained music listener, I tend to describe music using phrases like the above. But if I were a songwriter or a composer, I might use specific terms like timbre, key, time signature, tempo, and others.

The text prompts that people use might also reference cultural figures that are unassociated with, or were briefly associated with music, such as “A pop song in the style of William Hung”.

The creators of a prompt-based music generator would need to make a product decision about whether or not to support natural language understanding (NLU) interface on top of just the features that the model was trained on, or also on a large and growing collection of cultural references that people might use to describe a sound.

Supporting cultural references would make a tool more fun to use, and perhaps offer stylistic shortcuts for people writing prompts, but would also require extensive cultural metadata to use as training data.

Verbal descriptions of music are inherently lossy. In choosing what to describe, you leave something out. I might choose to describe the vocal qualities of a track that I want to hear and neglect to mention anything about the rhythmic qualities: “bright vocals sung by a feminine voice and a guitar solo in the middle” could produce a nice track with an unintended trap-style backbeat.

Even when you’re being specific, it’s possible for your description to be lossy. For example, If you’re trying to describe Godford, you might say “they sound like The Blaze”, but that only works for someone who has listened to The Blaze (or in this case, for a model that has been trained on songs by The Blaze). Also, it only describes one specific song, not the whole vibe of the artist.

In this case, if you were to prompt a generative music model with: “Short vocal track with flute that sounds like Godford”, would the model produce a song more like The Beast or Saw You?

Words are also subjective. I was explaining to a friend that I thought HAAi’s DJ set at a recent festival was darker than I expected—but my friend didn’t know how to interpret my description. It was clear to me that a dark set meant that it was bass-heavy and grime-influenced, but darker can also mean emotionally darker, not just darker in timbre.

As an artist named Emmanuel describes in Why Dall-E will not steal my job as an illustrator, documenting his experience creating DALL-E 2 prompts:

“I forgot to mention so many details that seemed intuitive for me, like having the characters smile, looking in front of them, having the woman driving, the characters having their seatbelts on etc…”

As such, a text prompt might have to be frustratingly long to be sufficiently descriptive in order to produce a quality music sample.

Therein lies the worst part of using words to describe music, which people using text-based image generators already encounter: it’s difficult to “tune” text input in a way that produces your desired output.

It’s difficult to use text to fine-tune the output

When we’re prompting generators, we usually have an expectation in mind of what the output will look like (or in this case, sound like). Sometimes the surprise of an unexpected result can lead to excitement or a creative inspiration, but it can also be frustrating if you’re trying to iterate on a specific concept.

Like the illustrator Emmanuel explained in his blog post, Why Dall-E will not steal my job as an illustrator, the inability to definitively specify or lock certain qualities of the output images made it difficult to imagine such a tool being used to replace storyboarding or advertising campaigns.

In order to supplant some of these usability challenges, you’d need some pretty high guardrails on the types of terminology people could use, and some suggestions for how to phrase prompts, in order to help people write useful prompts from the start.

You’d also want to give people the ability to refine a given output. If you wanted to “save” parts of the generated output and re-generate other parts, the model would need to have a high level of explainability to where you could use text to describe what needed to change on a regeneration and the model could do that.

Barring a level of interactivity with the model to selectively regenerate portions of the track, a prompt-based music generator would need to be capable of exporting the generated music as stems, or the requisite parts of the track (vocals, drums, etc.) for exploration and adjustment in a music composition and production tool like Apple Logic Pro or Ableton Live.

Music generation needs to be fast

A final usability challenge is the time it might take to generate the output.

The OpenAI blog post section on limitations brought this challenge to my attention:

It takes approximately 9 hours to fully render one minute of audio through our models, and thus they cannot yet be used in interactive applications.

In order for a music generation tool to be usable and interactive, it would have to generate music in a few minutes at most—not 9 hours.

Quality and ethical challenges

Generating music from a text-based prompt input is difficult to make usable, and technically challenging, but there are also quality challenges involved.

Quickly created music probably sounds bad

Concept artist RJ Palmer, as quoted by Luke Plunkett for AI Creating ‘Art’ Is An Ethical And Copyright Nightmare, points out about AI image generation:

The unfortunate reality of this industry is that speed is favoured over quality so often that a cleaned up, ‘good enough’ AI-generated image could suffice for a lot of needs.”

For many of the commoditizable uses of music, such as background music for YouTube videos or podcasts, availability and affordability often trump quality. The types of shortcuts that easily generated music could lead to might reduce the quality of music that people encounter day-to-day.

It’s entirely likely that those types of shortcuts are already being taken when producing that kind of commercial music, but additional quality challenges are specific to machine-learning-generated music.

Generated music might all sound the same

Christian Thorsberg³ interviewed Patricia Alessandrini, a composer, sound artist and researcher at Stanford University’s Center for Computer Research in Music and Acoustics. She pointed out this key flaw in using machine learning to generate music:

AI is better at generating content faster and copying genres, though unable to innovate new ones — a result of training their computing models, largely, using what music already exists.

We already see this with music to a certain degree, as Sean Michaels reports for The Guardian in Pop music these days: it all sounds the same, survey reveals.

Data analysis and algorithms are only adding to this effect, as Derek Thompson reports for the Atlantic in The Shazam Effect.

The increasing sameness of music can be attributed to record labels investing in “safe” artists that sound similar to acts already succeeding on the market, a desire to work with the same hot producers (Jack Antonoff, Max Martin, etc.), or even a music marketing problem where only familiar sounds get marketed in reliable markets.

Those decisions are often more visible on the top of the charts, where the network effects come together, but a machine learning model generating music would also be facing down similar problems on a smaller scale. The model would also have to produce novel sounds frequently enough to not be stale.

Much like how recommendation systems can lose listeners if they start recommending tracks that start to sound the same, if the music generating model learns a set roster of go-to sounds or beats, then the output might be more easily identifiable as artificially generated, repetitive, and uninteresting.

As v buckenham points out on Twitter⁴:

there is a real fear among AI researchers that the last big corpuses of human written text have already been captured. all future scrapes of the internet for text to learn from will be contaminated by machine-speak.

A prompt-based music generator could lead to a flattening of newness in mainstream music. In fact, we might already be headed there given the number of generative music tools already on the market.

Swedish artist Simon Stålenhag summarizes the cultural threat on Twitter⁵:

“What I don’t like about AI tech is not that it can produce brand new 70s rock hits like ‘Keep On Glowing, You Mad Jewel’ by Fink Ployd, but how it reveals that that kind of derivative, generated goo is what our new tech lords are hoping to feed us in their vision of the future”.

Further contributing to the likelihood of sameness driven by machine learning feedback loops and training data contaminated by generated music is the fact that ML engineers can’t dig their way out of this challenge by collecting even more data.

More data doesn’t lead to higher quality music

As Dr. Timnit Gebru points out in a Twitter thread about Stable Diffusion:

“Size doesn’t guarantee diversity”⁶

Dr. Gebru is referring to training large language models and the fact that scraping large volumes of text from the web only captures what people have written in English on openly accessible webpages—thus ignoring anyone that writes in a different language, is unable to communicate in writing, or who chooses to communicate only in closed, unscrapeable online spaces such as WhatsApp groups or Signal chats.

Similarly, if a generative music model is trained only using the Million Songs Dataset, or even the music available to existing digital streaming providers (DSPs), then only artists with digitized music, and with the means to upload their music to DSPs and distribute it with accurate explicit metadata can be represented in a generative music model like the one I’m outlining in this post.

While it’s possible to use a diffusion model to build a music generator without extensive music metadata, a prompt-based music generator would need to use more curated datasets instead of audio files scraped off the internet in order to make the semantic connections between the text of a prompt and the expected audio features.

This means that the only music available to train a prompt-based music generator is digitized music, available on the internet, likely already licensed to DSPs, and with ample metadata available.

Legal challenges

As long as we’re talking about indiscriminately scraping data off the internet, it’s relevant to talk about the legal challenges inherent in training and releasing a prompt-based music generator.

I’m an armchair speculator, not a lawyer. I have no legal training, so don’t take any of this for legal advice.

Can you train a machine learning model with copyrighted music?
Do you owe music royalties if you train a machine learning model on copyrighted music?
What can you do with the music that you generate with such a tool?

Is it legal to train a machine learning model on copyrighted music?

Is it legal to train models on copyrighted music? And even if it is, would you owe royalties to anyone after doing so?

Issues with copyright have already come up with prompt-based image generators. Unlike image copyright, music copyright and royalties are much more complicated.

This overview from the United States Congressional Research Service⁷ outlines some of the complexity:

Songwriters and recording artists are generally entitled to receive compensation for (1) reproductions, distributions, and public performances of the notes and lyrics they create (the musical works), as well as (2) reproductions, distributions, and certain digital public performances of the recorded sound of their voices combined with instruments (the sound recordings). The amount they receive, as well as their control over their music, depends on market forces, contracts between a variety of private-sector entities, and laws governing copyright and competition policy. Who pays whom, as well as who can sue whom for copyright infringement, depends in part on the mode of listening to music.

Reproduction, distribution, and performance of music are the key components of music copyright. The work, reproduction of the work, and the written score used to produce the work are each separately licensed, and royalty formulas also consider how the work was distributed—as physical media, through interactive streaming, or noninteractive streaming.

There are different methods that you can use to train machine learning models, and it’s unclear whether the different methods would be treated differently under copyright law.

For example, models like the symbolic generators described in the section on Music generation is hard would be trained with computed representations of music, where metadata describes the audio in alphanumeric terms.

Other models are trained by having a model “listen” to music, identify patterns, and attempt to recreate those patterns. This is the method that diffusion models, like Stability AI startup HarmonAI’s tool Dance Diffusion use.

It’s unclear whether these methods would be considered a form of reproduction, distribution, or performance of the music, or instead considered a form of data mining for the purposes of copyright law.

For one perspective on the copyright law, the Recording Industry Association of America (RIAA) thinks that using copyrighted music to train machine learning models is a form of copyright infringement, stating in a comment to the Office of the United States Trade Representative⁸ that:

To the extent these services, or their partners, are training their AI models using our members’ music, that use is unauthorized and infringes our members’ rights by making unauthorized copies of our members works.

However, the RIAA made this comment about tools that use machine learning to break apart a music track into stems, rather than music generators specifically.

Depending on the influence of the RIAA and how the use of AI-based music generation evolves in the songwriting and production portion of the music industry, determining the rights and royalties held by music generation tools could become another layer on an already-complex layer cake of music rights.

Depending on the method used to train the machine learning model, the training dataset might infringe copyright and might not. But copyright law doesn’t only exist in the United States.

The United Kingdom has a provision that makes a copyright exception⁹ for the following:

Text and data mining is the use of automated analytical techniques to analyse text and data for patterns, trends and other useful information. Text and data mining usually requires copying of the work to be analysed.

An exception to copyright exists which allows researchers to make copies of any copyright material for the purpose of computational analysis if they already have the right to read the work (that is, they have ‘lawful access’ to the work). This exception only permits the making of copies for the purpose of text and data mining for non-commercial research.

This exception is for non-commercial research, but the UK government plans¹⁰ to expand the copyright exception for text and data mining (TDM), concluding that:

For text and data mining, we [the Government of the UK] plan to introduce a new copyright and database exception which allows TDM for any purpose. Rights holders will still have safeguards to protect their content, including a requirement for lawful access.

It’s unclear what those safeguards and lawful access requirement will look like in practice. The UK isn’t the only country with this type of exception, however. As part of the reasoning that the UK government offers for allowing this exception for text and data mining, the document points out:

Several other countries have introduced copyright exceptions for TDM. These encourage AI development and other services to locate there. Territories with exceptions include the EU, Japan and Singapore. TDM may also be fair use under US law, depending on the facts.

There’s clearly precedent for allowing text and data mining, which can produce datasets to train machine learning models, in the EU, Japan, Singapore, and soon the UK. Expanding this exception would help the organizations I identified that could build a prompt-based music generator.

Even without explicitly permitting commercial use in copyright law, companies still find a way to use datasets created only for non-commercial use. As investigated and named by Andy Baio, AI data laundering provides a loophole:

It’s become standard practice for technology companies working with AI to commercially use datasets and models collected and trained by non-commercial research entities like universities or non-profits.

While United States copyright law might be unclear about the legality of training a machine learning model on copyrighted music, the United Kingdom, Japan, Singapore, and countries in the European Union all seem to permit it. Even without that express permission, AI data laundering is already happening to permit it anyway.

What about fair use?

Copyright exceptions and AI data laundering help organizations circumvent copyright restrictions for training data models. When it comes to the output produced by the generative models, the concept of fair use might come into play for United States copyright cases.

Fair use is something that needs to be decided on a case-by-case basis by the courts, but the ruling in “Authors Guild v. Google” sets a helpful precedent. In this case, the court concluded that¹¹:

The purpose of the copying is highly transformative, the public display of text is limited, and the revelations do not provide a significant market substitute for the protected aspects of the originals.

If you’re training machine learning models with copyrighted music with the intent of generating music that would then almost certainly “provide a significant market substitute” for original works of music, I’m not certain you’d succeed in court with a US copyright fair use claim for the generated music.

As a workaround, you could argue that the generated music functions as a derivative work rather than a replacement for the original work. And derivatives aren’t unique to generated music.

Claire Evans, writer and member of YACHT, points out in an interview¹² that music creation without prompt-based music generators is also often derivative:

“That’s something that artists are already doing in the studio in a much more informal and sloppy way,” Evans said. “You sit down to write a song and you’re like, I want a Fall bass line and a B-52’s melody, and I want it to sound like it came from London in 1977.”

But while visual artists might not be able to do much about a lookalike image, musical artists already have copyright protections for various types of derivatives, including direct samples of their work as well as remaking part of a song, called interpolation.

This complexity means that any “soundalike” clips generated by prompt-based generative model, even if you’re not knowingly, intentionally, or actually sampling a musician’s work, you might owe an interpolation credit to some entity.

Unlike Taylor Swift, who filed a motion in a copyright lawsuit claiming that she “had never heard the song Playas Gon’ Play and had never heard of that song or the group 3LW”, music created by a prompt-based music generator can’t use that defense unless the track being infringed on was not part of the original or subsequent training data for the generative model.

This risk is confirmed by Kyle Worrall, “a Ph.D. student at the University of York in the U.K. studying the musical applications of machine learning” in an interview with Kyle Wiggers and Amanda Silberling for TechCrunch¹² about the originality of generated output:

“There is little work into establishing how original the output of generative algorithms are, so the use of generative music in advertisements and other projects still runs the risk of accidentally infringing on copyright and as such damaging the property,” Worrall said. “This area needs to be further researched.”

So while the use of copyrighted music to train a machine learning model might be protected by a copyright exception, and possibly even fair use, the generated music produced by the model could still infringe on copyright and require royalties to be paid.

What can you legally do with generated music?

Given this copyright context, it matters what you do with the output from a prompt-based music generator.

As pointed out in the report from the United States Congressional Research Service⁷:

Songwriters and recording artists are generally entitled to receive compensation for (1) reproductions, distributions, and public performances of the notes and lyrics they create (the musical works), as well as (2) reproductions, distributions, and certain digital public performances of the recorded sound of their voices combined with instruments (the sound recordings).

If you generate music that might infringe on the copyright of songwriters and recording artists, you might owe royalties if you reproduce, distribute, or perform that music (live, on the radio, or through a streaming service, to name a few examples).

The main goal of using a prompt-based music generator is to create royalty-free music for specific use cases, but copyright infringement risks might mean you owe royalties anyway.

If you wanted to hedge those copyright infringement risks by copyrighting the generated music yourself, then you encounter more legal challenges.

Royalties can be allocated to songwriters, producers, vocalists, musicians, and others involved in making a track. If some or part of the track is the result of a prompt-based music generator, what happens to that share of the royalties?

Furthermore, to whom would those royalties get paid out?

The people that trained the machine learning model that generated the music?
The people that wrote the machine learning model?
The people that prompted the model to generate that specific track?

It probably doesn’t matter because it’s likely that using a prompt-based music generator (or even the existing music generation tools today) could prevent you from copyrighting tracks created that way.

In the United Kingdom, copyright law protects generated works for 50 years, with the authorship described as follows¹³:

In the case of a literary, dramatic, musical or artistic work which is computer generated, the author shall be taken to be the person by whom the arrangements necessary for the creation of the work are undertaken.

The same effort that resulted in plans to expand the copyright exception to text and data mining made no changes to this law, asserting¹⁰:

For computer-generated works, we plan no changes to the law. There is no evidence at present that protection for CGWs is harmful, and the use of AI is still in its early stages.

So you can copyright your generated music tracks in the UK. In the United States, however, it’s more complicated.

Luke Plunkett in AI Creating ‘Art’ Is An Ethical And Copyright Nightmare identified a copyright case that is likely to have ramifications for music copyright in the United States if music generation becomes more common. To quote his article:

In February, the US Copyright Office “refused to grant a copyright” for a piece of art made by AI, saying that “human authorship is a prerequisite to copyright protection”. That case is now being appealed to a federal court, however, because the AI’s creator thinks that, having programmed the machine, he should be able to claim copyright over the works it produces. Even when a decision is ultimately reached in this case, it will take a lot more time and cases for a firmer legal consensus to form around the subject.

But what is that work the AI’s creator is claiming, if not simply a casserole made from art created by actual human artists, who are not being paid or even credited for their contributions?

Therein lies the challenge with granting copyright on “computer-generated works”—which person has the right to claim authorship? You might not be able to copyright the tracks you create with a prompt-based music generator at all.

There’s a lot of copyright law to untangle for prompt-based music generators, from the training and development perspective, as well as the distribution, copyright infringement risks, and challenges with getting your own copyright for the tracks if you want to.

The legal challenges are perhaps the most complex, but not the only challenges facing anyone trying to build a prompt-based music generator. Given that, will we see one at all?

Will we see a prompt-based music generator?

Given the market opportunity for generated music and the long list of challenges, will we see a prompt-based music generator?

It’s tough to say. If DALL-E 2 and Stable Diffusion survive past the “hype cycle” and are consistently used, then we’ll almost certainly see prompt-based music generators.

Getting high quality music from a prompt-based music generator is the biggest challenge.

To get there, two technical advancements are necessary:

Developing the machine learning methods necessary to generate high-quality music based on an arbitrary output.
Evolving the song-characteristic prompt used by many music generator tools already to use full natural language, cultural references and all.

The existing machine learning capabilities can do a lot, but we might end up with new types of training (like diffusion models) or elaborate iterations of training (like the “conditioning” that OpenAI used with Jukebox) before we get music generators that can emulate song structure.

If the machine learning methods get solved before the natural language prompt is wired up to audio generation, we might end up with an audio-based prompt before a text-based prompt.

Most people are musical to a degree, and there might be more novelty in humming a ditty into your laptop microphone and getting a 15 second audio clip than there is in entering a string of text into a text box and trying to get it to output something that sounds like what you can hear in your head.

With that audio-based prompt, you can also avoid many of the usability challenges with using text to describe music.

It’s tough to say how soon we’ll see a prompt-based music generator. While HarmonAI is making progress with Dance Diffusion, any pre-trained models they can release are limited by the availability and quality of music datasets.

Even if we do see a prompt-based music generator, it might not be consumer-facing. For now, it seems like companies that might have a market advantage developing prompt-based music generation are investing elsewhere.

The companies that I identified in Who could build a prompt-based music generator all use machine learning extensively, but for applications of ML that make tasks easier, increase engagement, and make the companies more money.

For example, the following ML applications:

Adding computer vision capabilities to photos at Apple
Improving music track isolation for Shazam at Apple
Separating tracks into stems using machine learning at Spotify
Improving podcast discovery at Spotify
Improving recommendations and personalization for Radio at Pandora

Out of any of these, Spotify is the most likely to invest in music generation.

In 2017, Spotify hired François Pachet to direct “the Spotify Creator Technology Research Lab, where he designs the next generation of AI-based tools for musicians.”, as reported by Music Business Worldwide.

We could see some more AI-based tooling from Spotify’s music creation research lab, but given Spotify’s increasing investment in podcasts, I think it’s most likely that the first iteration of a music generation tool would be for internal use to support podcast creators with the type of commercialized audio that functions well as intro, outro, and background music in podcasts.

What might we see before a prompt-based music generator?

We’ll continue to see evolutions of the music generation tools already on the market. Those exist as collaborators to music producers, songwriters, and other musical artists.

Perhaps we’ll see SoundCloud release some machine learning-based tooling for musicians innovating on their service, or tools exclusive to Warner Music Group to help their artists iterate faster on their songs—an increasing emphasis on producing music efficiently while also claiming to retain some elements of creativity.

Whether or not we see these changes at major labels or services that support entry-level artists depends on who is in power at these companies, and how much they favor the organic creative experience over an extended commoditization of media and music to maximize shareholder value.

If we do see a prompt-based music generator soon, especially one that doesn’t work very well, the text prompt data will be valuable for music industry players to evaluate. Record labels might want to analyze the types of prompts entered into the service, especially in early adoption stages before people learn the “prompt engineering tricks” to produce specific sounds from the model, rather than describe what they want to hear.

The text prompt data could provide inspiration or guidance to the agents and others at a record label trying to identify the next big sound, or surface an underserved niche in the music market.

Music generation generally also offers the possibility of generating an audio clip to test an idea or a music market by using that clip in a TikTok or an Instagram Reel. If the video goes viral, then you could attempt to generate a full track, or seek out artists with that sound to produce a full track inspired by the audio clip.

Of course, virality doesn’t necessarily lead to success, as the TikTok artists profiled in the Vox Earworm video with The Pudding, We tracked what happens after TikTok songs go viral reveal, so this isn’t a foolproof method of testing the market, but it is a way to try to shortcut finding the next big hit.

No matter what happens with music generation tools, musicians aren’t going anywhere. As musician and YouTube personality Adam Neely points out in his video Will AI replace human musicians? | Q+A, technology replacing musicians has been a fear for decades—but we’ll all keep making music nonetheless.

Thanks to EI d’AU, Youri Tjang, and Aleksandra Lazovic for your Excalidraw libraries, as well as Clay and Thomas in Discord for providing helpful feedback on earlier drafts of this post.

The current version of capitalism says yes. ↩︎
Linked by the Weights and Biases article, A Gentle Introduction to Dance Diffusion. ↩︎
Reporting for Grid in The future of AI in music is now. Artificial Intelligence was in the music industry long before FN Meka, linked to me by the Platform & Stream newsletter. ↩︎
Thanks to Assaf Arkin’s weekly Labnotes newsletter. ↩︎
Referenced by Luke Plunkett in AI Creating ‘Art’ Is An Ethical And Copyright Nightmare ↩︎
Gebru is referencing her paper, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? with Emily Bender, Angelina McMillan-Major, and Shmargaret Schmitchell. ↩︎
Quote from page 4 of version 22 of report R43984 from the Congressional Research Service entitled “Money for Something: Music Licensing in the 21st Century”. ↩︎ ↩︎
Quote from Page 12 and 13 of RIAA Submission to Comment Request for the 2022 Review of Notorious Markets for Counterfeiting and Piracy, uploaded and shared in an article by Ernesto Van der Sar on Torrentfreak titled RIAA Flags ‘Artificial Intelligence’ Music Mixer as Emerging Copyright Threat. The comment request the RIAA is responding to is 2022 Review of Notorious Markets for Counterfeiting and Piracy: Comment Request. ↩︎
Text and data mining for non-commercial research on the Exceptions to Copyright page for GOV.UK. I found out about this exception in UK copyright thanks to this tweet by Kevin Brennan MP. ↩︎
Artificial Intelligence and Intellectual Property: copyright and patents: Government response to consultation. ↩︎ ↩︎
Page 230 of Authors Guild v. Google, Inc., 804 F.3d 202 (2015) decision by the United States Court of Appeals for the Second Circuit. Thanks to Andy Baio in his blog post about copyright and AI tools, With questionable copyright claim, Jay-Z orders deepfake audio parodies off YouTube, for transcribing the video by the anonymous creator of Vocal Synthesis, Barack Obama and Donald Trump read a special message from this channel’s creator (Speech Synthesis), which led me to discover this case. ↩︎
Interviewed by Kyle Wiggers and Amanda Silberling for TechCrunch for AI music generators could be a boon for artists — but also problematic. ↩︎ ↩︎
Page 32 of the Copyright, Designs and Patents Act 1988, in the PDF linked to from that page. ↩︎

Tips for writing SaaS documentation

Wed, 26 Oct 2022 22:17:57 +0000

If you’re writing documentation for a software-as-a-service (SaaS) product that releases constantly, it’s easy to be overwhelmed by the amount of new functionality being released that needs to be documented.

But do you really need to document all of it? Nope.

Instead, document the hidden, document the weird, and document the why.

Consider every new product feature in terms of what the user cares about. Don’t document what the product can do—document what the user wants to do.

I recommend adopting minimalism in your documentation, especially if you’re trying to keep up with the fast-paced release cycle of a SaaS product. Anni Bond wrote an excellent overview on Adopting minimalism in your docs.

I use a series of questions to keep my documentation minimal and to focus my documentation efforts.

Documenting a feature

When reviewing a feature, consider the following:

What might confuse someone about this?
What could go wrong that they’d need help fixing?
What is different enough from existing product experiences that might require shaping a mental model?
Where do these changes fit in a customer workflow?
In what ways could the feature be improved to be clearer or easier to use (without documentation)?

It’s often helpful to use your role as a user advocate to push for product improvements rather than documenting around problems with the user experience.

Reviewing doc feedback

When reviewing documentation feedback from a customer or the field, consider the following:

Was more than one person confused?
Is this feedback related to one specific edge case?
Is this an actual error? How much time will it take to track down?
What led to someone being confused? Is their personal mental model different?
Is it actually confusion about how the product works and isn’t necessarily a documentation task?

Try to understand the root of the feedback. Rather than giving in and adding a quick note about anything and everything, consider what improvements might make sense to make to the documentation or the product to address the core issue.

Documenting strange behavior

When reviewing a request from PM or engineering to document something weird, consider the following:

Is this time-limited?
- Is this behavior only going to be true until a certain point of time?
Is this feature-limited?
- Is this behavior only going to be true until a feature gets refactored/improved/enhanced?
- Is that work scoped and planned for anytime soon?
Is this actually a bug that should be in the release notes and nowhere else?
How many people are affected by the weird thing?
How common or likely is the weird thing to occur in actual customer environments?

Write less for success

It might seem like I’m telling you to find a lot of reasons to avoid writing documentation. I am. It’s important to focus on the documentation that customers are going to go looking for, instead of making sure the entire product functionality is documented.

Say your product requires people to confirm their email address before they can start using it.

Will customers read documentation about how to confirm their email address? No.

Do you need to write documentation about how to confirm an email address? Also no.

If you have a strong understanding of your users and their mental models, you can make this same determination for other, more complex, workflows in your product.

Stop trying to keep up with the product and start helping the people using it.

Technical documentation as a map

Mon, 19 Sep 2022 20:46:08 +0000

Matt Webb wrote a post about organizing data and mapping the experience of the web, and that made me consider how the decisions about documentation structure, especially in the early stages, require similar decisions.

Technical documentation functions as a map for your product. For specific users, your documentation provides wayfinding guidance and provides the information necessary to navigate the space relevant to your product.

Beyond a map for your product, technical documentation can also map out specific workflows within your product—specific routes, one might say.

For example, diagramming and describing how data flows through the 55+ different systems that comprise Facebook. Without that documentation, you might find yourself in the same situation as some Meta engineers did in March¹, admitting that:

“we have a somewhat strange engineering culture compared to most where we don’t generate a lot of artifacts during the engineering process. Effectively the code is its own design document often.” If the code serves as the documentation, that’s the same as having street signs or individual transit stations, but no sense of where you are within a larger system. You need a map to add a layer of intelligibility to the individual parts of an overall system.

Considering your documentation as a map can help you when you’re struggling to know where to start when writing minimum viable documentation.

What kind of map might you draw on a napkin about your product, or about a specific data interaction or user interaction?

What is the key wayfinding information? What are the intersections or landmarks to be aware of?
What would confuse someone if you left it out of the map? Are your steps complete enough to help someone get to their desired destination?
Do you understand the destination and the person you’re guiding well enough to draw such a map?

If you consider those questions the next time you need to write documentation about a product, a data flow, or a user flow, you and your customers will be much better situated.

As disclosed in a court filing reported by The Intercept in Facebook Engineers: We Have No Idea Where We Keep All Your Personal Data and further reported on by Vice in Facebook Engineers Admit They Don’t Know What They Do With Your Data. ↩︎

Write better docs with a product thinking mindset

Mon, 25 Jul 2022 15:30:00 +0000

I’ve frequently seen product thinking discussed in product management and user experience design contexts, but haven’t seen it applied to technical writing and documentation. And yet, by applying product thinking to documentation, we can write more useful, relevant, high quality documentation.

What is product thinking?

Product management thought leader Shreyas Doshi defines product thinking as follows in a Twitter thread:

“Product Thinking is about understanding motivations, conceiving solutions, simulating their effects, and picking a path based on the effects you want to create.”

As a technical writer, I focus on user problems and the customer journey as a central part of my job, so it seems obvious to adopt product thinking for documentation. While I was working at Splunk, the documentation culture fostered a product thinking mindset, enshrined in the Splunk docs book, The Product is Docs.

Despite being immersed in that culture and caring deeply about helping users, I still found it easy to fall into a “just get it done” mindset, or what Doshi calls project thinking:

“Project Thinking is about understanding expectations, formulating plans, marshaling resources, and coordinating actions to meet those expectations.”

Where project thinking focuses on the steps and process for creating a product, product thinking focuses on the motivation for creating a product and the possible outcomes of that motivation.

Why project thinking is pervasive in technical writing

Technical writers are often short on time and resources. Forced to keep track of multiple teams and projects, we often spend what energy we can on the priorities that seem most urgent. Even with sufficient resources and planning, time crunches still happen. With these limitations, it’s easy to focus solely on churning out documentation and just keeping up.

Sprinting alongside engineers, we often adopt engineering practices like breaking our work down into small parts, writing tickets and adding them to sprints to track with developers. With the widespread adoption of CI/CD pipelines in product development, these practices seep further into our documentation processes.

It feels like an optimization shortcut to consider documentation projects in terms of “feature coverage”, much like how quality assurance projects are considered in terms of “test coverage”. Documentation can become another box to tick, making sure that every pull request deployed has sufficient test coverage and documentation coverage for the product functionality.

This “coverage” mindset is ruled by project thinking. What if we took a product thinking approach?

Project thinking vs Product thinking

While project thinking leads us to write documentation that ensures coverage a specific product feature, product thinking helps us focus on what needs to be written and why. Embracing product thinking lets us write higher quality documentation that is more relevant to customers.

That probably seems too good to be true, so let’s look at a specific example—a product making changes to the API endpoints available to developers as part of a versioning change.

Project thinking for API endpoint deprecation

Taking a project thinking approach to the documentation might look like the following:

documenting each new endpoint
making sure each deprecated endpoint has a link to the relevant new endpoint

There might be also be a deprecation plan in the project that includes “Add a banner to the old endpoints with a link to the new endpoints indicating that they are deprecated and will be removed in the future”.

As a project thinker, this approach has the essentials:

Stage	Action
Expectations	Document the new API
Planning	Docs and engineering work together to produce updated documentation Provide deprecation information to users
Resources	Docs and engineering
Delivered Output	Documentation available with new API release and shared deprecation plan

This approach focuses primarily on coverage for the new functionality, and offers some form of continuity for users using the old functionality by alerting them to the change.

Product thinking for API endpoint deprecation

Taking a product thinking approach to the documentation might instead look like the following:

As soon as you know API endpoints will be changing, write learning outcomes such as the following examples:
- As a developer, after reading the API docs I can successfully update my code to use the new endpoints.
- As a developer, after reading the API docs I am confident that I won’t lose any data and am correctly handling data formats sent by the new endpoints.
- As a developer, after reading the API docs I can explain to my colleague who was on vacation which endpoints changed.
- As a developer, I want to explain to my boss when I need to deploy my new script (when the API changes become permanent).
Documenting each new endpoint.
Making sure each deprecated endpoint has a link to the migration guide.
Writing a guide for users of the v1 API to help them migrate to the v2 API, that addresses the specific learning outcomes you defined by including the following:
- A note or caution that calls out any changes to data formats.
- Specific code examples comparing API calls using the deprecated endpoints with the API calls using the equivalent new endpoints and parameters.
- Clear dates or communication methods where developers can find out when the v1 API stops working.

From a product thinking perspective, this list of steps covers the essentials:

Stage	Action
Motivations	Identify the customer motivations by writing learning outcomes
Solutions	Make a list of documentation options that address customer motivations.
Simulations	Write a migration guide and draft specific code examples to validate with engineering and product management.
Desired Outcomes	Customer achieves business continuity and engineering can deprecate old API endpoints and stop maintaining the code.

This approach focuses primarily on customer outcomes, rather than optimizing for producing a specific type of documentation or specific output that can be checked off a plan.

Comparing the results

If project thinking optimizes for coverage and output, product thinking optimizes for outcomes. The documentation written from a product thinking mindset seems like it’s a lot more detailed, but also slow and resource intensive to create. And it is.

But product thinking is only slow in the short term. If you focus on coverage (and output) instead of outcomes (and customer success), you might have complete documentation, but it’s not as useful to customers. Over time, you might not have many of those to worry about.

I’m not advocating for abandoning project thinking entirely. We can’t neglect resource considerations or the content delivery timing of what we’re writing, especially if we’re working as lone writers. But just as it’s crucial to consider the timing of what we write, we need to consider the usefulness of what we write. Cultivating a product thinking mindset is essential to writing useful documentation.

Applying product thinking to documentation

It’s easy to fall into the habit of being a “reactive” documentarian, responding to requests for information and rotely updating page after page after page with more information. As technical writers, our priority lists are often shifting or shuffling around. So how do you go about applying product thinking to your documentation?

Approach the documentation itself like a product.

Create cues in your writing process, get access to people with essential information, write learning outcomes for your documentation, consider the content design of your documentation, and test everything about your proposed changes!

Create process cues

To make sure you don’t spend too much time on “getting the writing done”, add cues to your documentation process that can help you maintain user-centricity in your writing. What kind of cues? It depends on how you do your writing, but here are some that have worked for me:

Follow a template for documentation tickets. On one fast-paced development project, I made sure that every documentation ticket I filed included the following:

the audience of the content
the technical SMEs that knew more about the feature
the learning outcome(s) for the content

If I couldn’t fill out those details, I knew I couldn’t work on the documentation request without producing poor quality content.

Review and plan your work regularly. On another project, I embraced the essentials of project thinking by spending some time on Friday afternoons reviewing what I’d accomplished in the last week and planning out my tasks for the following week. An output-focused approach that I combined with an outcomes-oriented approach by identifying the key product outcomes or customer goals associated with each documentation task I planned to work on.

Get access to people

It’s impossible to write product-focused documentation with customer outcomes in mind if you don’t have the access to the product managers, engineers, and customers to understand and validate those outcomes.

Make sure you can talk to the product managers, engineers, and UX designers working on your product, but especially take steps to build relationships with sales and customer support representatives that interact directly with customers every day.

With access to product experts and customer experts, you can more readily identify and validate the customer motivations driving the development of specific product functionality. This is especially crucial in an agile development environment, where the functionality that you’re documenting might only be a small building block of a larger solution, but in isolation it seems clunky and broken. For example, an incident management feature that adds the ability to store information about an incident, but neglects a status field to denote the state of the incident.

With a deep understanding of the why and why now behind product development decisions, you can structure and write your content in a way that accommodates customer goals alongside these possible product limitations. For example, you can write an incident management workflow that includes options for adding a status to an incident. In this way, you’re writing documentation that addresses the existing feature, the customer’s desired outcome, and leaves room to showcase future product functionality.

Write learning outcomes

It’s crucial to understand the customer’s desired outcome when using a product. As the technical writer for a product, however, you also want to establish the customer’s outcomes when using your documentation.

When you write a learning outcome, you want to use active words and be specific. These aren’t abstract user stories, but rather concrete outcomes that customers can achieve.

Don’t write this:

After reading the docs, I understand how to call the API.
After reading the docs, I understand how to create an incident in the product.

Instead, be specific about the desired customer outcomes:

After reading the docs, I can successfully update my code to call the new API endpoint
After reading the docs, I can confidently create an incident based on a phone call from DevOps.

You need to write specific learning outcomes for your documentation so that your documentation can also be specific and useful to customers.

If you can’t write a compelling and specific learning outcome for a feature, several things could be true:

You might not need to document it. If there aren’t any associated customer outcomes, it probably doesn’t need documentation.
You might not understand the customer motivations well enough. Check in with product management and the technical SMEs for the feature to dig into why the feature is being added to the product to uncover the motivations driving development.

Writing learning outcomes for every piece of documentation you produce (including the small ones) helps shift you to a product thinking mindset and reminds you why your documentation is important.

Consider content design

Another element of the product thinking mindset is considering new ideas. Content creation can feel routine and straightforward, but it involves design just as much as it involves writing.

Considering the content design of your writing can make your writing more accessible, more inclusive, and more useful. If you include more headers or change paragraphs to a table or a list, your content is easier to scan and therefore more helpful to anyone easily bored, in a hurry, and with dyslexia and/or ADD.

You can also break up paragraphs with visuals such as diagrams or screenshots, with helpful alt text, which can reinforce your content and provide extra learning opportunities for more visual learners.

In addition to adjusting the design of your text, you can also change how and where you present your content. Consider the following alternatives to text documentation:

An interactive product tour
A blog post
A 2-3 minute video
An interactive tutorial that uses a demo environment

For example, you could write a blog post to walk through incident management functionality in your product using a common IT incident or a security exploit from the news. This alternate form of content provides a tangible relevant example to customers in a dated blog post that you won’t have to keep updated.

Test everything

To continue treating documentation like a product, you need to test out the decisions that you make along the way. Validate the product motivations you’ve identified, the customer learning outcomes you’ve defined, and take advantage of any and all testing options available to you.

Get a review from technical SMEs about the accuracy of your content.
Ask a peer for feedback on the presentation of your content, especially if you’re trying something new.
Send your writing to get edited, if you have the luxury of access to an editor.
Check your writing for style guide adherence and completion, using a tool like Vale or similar. I’m guilty of publishing docs with broken links and unfinished sentences. Automated or manual checklists are crucial to make sure your work is finished.
Perform UX testing on the content. If you restructured the information architecture, use tree testing to test out the intuitiveness of your changes.

Write better docs with a product thinking mindset

With a product thinking mindset, you build a clear understanding of why you are documenting something. In the end, that clear understanding produces documentation that provides a valuable resource to customers and can even prove to be a differentiator from your competitors.

2021 in Music: Spotify Wrapped, Last.fm, and Ethical Music Consumption

Wed, 15 Dec 2021 17:24:00 +0000

It’s been another unexpected year. Thinking about how I had a January and February stacked up the same way I spent all of my 2019, and booked up an April 2020 to be more of the same, and then whomp, pandemic, and all of my priorities changed.

I’m back with my yearly music data rundown, featuring a comparison with my Spotify Wrapped data (as always), a deep dive into my artist obsession of the year, some added stats about when I listen to music this year (and whether it’s different from past years), and more.

Want to skip around this post? Here’s what I cover:

What’s a musical aura?

Spotify Wrapped this year included an “audio aura” with my top “music moods”:

Digging into this feature, it seems that Spotify consulted an “aura reader” to assign colors to “six mood descriptor categories” and matched them with Spotify tracks.

After we created your personalized 2021 Wrapped experience, we assigned a weight percentage to the two audio “moods” that best represent your listening, and the final product is a swirling gradient of color. Below are the six moods and corresponding hues that could come up in your 2021 Wrapped Audio Aura.

My best guess is that there must be an internal API that contains mood descriptors for Spotify tracks that helps them build their emotion-based playlists. The closest metadata available in public APIs is the “energy” audio feature, which isn’t enough on its own to associate with an emotion (at least in my opinion).

I don’t yet have comprehensive Spotify audio feature data for my listening history, but I can’t begin to guess which music I listened to was categorized as “ecstatic” or “innovative”.

It’s purple and pink though, so that means:

Purple: This aura color pairs well with passionate music listeners. Purple auras tune in to get amped up, entertained, and moving while they navigate the twists and turns of their day.

Pink: Often described as the hopeless romantics of the aura spectrum, pinks view the world with a sense of optimistic, childlike wonder.

I wouldn’t describe my year, or even my music listening, as being largely driven by trying to stay amped up, entertained, occupied, optimistic, or wondrous about life. I felt fairly emotional and moody, tbh (those might be the same). If I take the time to dig into this further, I can check and see what the audio features are for my top Last.fm track:

and my top Spotify track:

Using the Spotify developer console to call the track audio features API yields the following metadata for Kyle (I Found You):

    {

      "danceability": 0.76,

      "energy": 0.669,

      "key": 5,

      "loudness": -9.105,

      "mode": 0,

      "speechiness": 0.285,

      "acousticness": 0.0757,

      "instrumentalness": 0.0215,

      "liveness": 0.77,

      "valence": 0.215,

      "tempo": 112.023,

      "type": "audio\_features",

      "id": "6Ao5d7TMQ92h87jQqSHGyw",

      "uri": "spotify:track:6Ao5d7TMQ92h87jQqSHGyw",

      "track\_href": "https://api.spotify.com/v1/tracks/6Ao5d7TMQ92h87jQqSHGyw",

      "analysis\_url": "https://api.spotify.com/v1/audio-analysis/6Ao5d7TMQ92h87jQqSHGyw",

      "duration\_ms": 196071,

      "time\_signature": 4

    }

And for Out Here With Us:

{

  "danceability": 0.695,

  "energy": 0.759,

  "key": 5,

  "loudness": -6.719,

  "mode": 0,

  "speechiness": 0.0346,

  "acousticness": 0.00856,

  "instrumentalness": 0.752,

  "liveness": 0.092,

  "valence": 0.589,

  "tempo": 109.983,

  "type": "audio\_features",

  "id": "6ezfPcUAN8UH6FaQbFfLSp",

  "uri": "spotify:track:6ezfPcUAN8UH6FaQbFfLSp",

  "track\_href": "https://api.spotify.com/v1/tracks/6ezfPcUAN8UH6FaQbFfLSp",

  "analysis\_url": "https://api.spotify.com/v1/audio-analysis/6ezfPcUAN8UH6FaQbFfLSp",

  "duration\_ms": 247955,

  "time\_signature": 4

}

Who knows how to really interpret these values, but here they are represented in a column chart, to prove to you that graphical representations of data are not sufficient communication.

It’s possible that the energy and/or valence of these tracks helped contribute to the music mood, but it’s honestly tough to say.

Spotify Wrapped: Unpacked

Spotify’s numbers of how many minutes I spent listening are pretty close to my less-accurately-calculated numbers, although as I pointed out last year, both numbers are fairly flawed.

(This has changed throughout the course of writing this post, because I listen to music while I write. If you see inconsistent data across screenshots, this is why. )

Because Last.fm doesn’t include track_length in their API, the best I can do is guesstimate according to the average length of a track in my iTunes library (though I was recently advised that it would make more sense to weight that average by tracks I actually listened to in 2021, which is correct and also more difficult math so I have not done this). I also factor in concerts and DJ sets.

My Last.fm totals are about 5,000 more minutes than Spotify estimates, although I have at least another month worth of data and 7 concerts to factor in. Neither of the numbers factor in any of the DJ sets I listened to offline in SoundCloud on my phone while I was flying around the country for Thanksgiving.

Lots of limitations to quantifying your life, it turns out! And I’m okay with that.

So that’s minutes listened, what else did Spotify tell me?

According to Spotify, my top songs are as follows:

Parra for Cuva - Out Here with Us
Parra for Cuva - Ordel
Jayda G - Both of Us (Edit)
Fred again.. - Dermot (See Yourself in My Eyes)
Fred again.. - Kyle (I Found You)

According to my Last.fm data to date, my “actual” top 5 songs are:

Fred again.. - Kyle (I Found You)
Fred again.. - Sabrina (I Am A Party)
Fred again.. - Marnie (Wish I Had U)
Fred again.. - Julia (Deep Diving)
Fred again.. - Dermot (See Yourself in My Eyes)

You might be sensing a theme here.

It’s worth pointing out that my Spotify top 5 are a bit better represented if you also consider the songs that I consistently listened to this year—namely, the tracks I listened to at least once a month this year. Considering that list, both Jayda G - Both of Us versions and Parra for Cuva’s track Out Here with Us are clear and consistent favorites of 2021.

Spotify also used some horrific graphic design to highlight my top genres of the year:

Deep house
Electronica
Indie rock
Vapor soul
Tech house

I included a bit of a discussion on genre in my post last year about Spotify Wrapped, and it seems that I’ve been meaning to dive into the meaninglessness of genre for awhile now. Suffice it to say, I don’t know what vapor soul is. I am still coming to terms with the fact that “tech house” is what I used to call “partyboi house”.

Fred again.. curiously has no genres listed on Spotify per the Developer Console:

But Parra for Cuva has several listed:

The other sources I have for genre, which is another word for “music category”, are Last.fm tags and iTunes genre for tracks that I’ve purchased this year. Last.fm offers up a fancy diagram of genres that I listened to since about June 2021:

I’m not totally sure what I was listening to in August 2021 that counted as Techno, but I’m glad it was short lived. It seems like Last.fm tags are either more generic, or have more overlap than other services, which is perhaps why Electronic, House, and Indie are so dominant compared to more descriptive genres. A very quick non-representative sample confirms this for me. Fred again.. is tagged as electronic, dance, hip hop, british, house, and personal favourites, making it easy to understand how electronic and house can both dominate the top of this chart.

Of course, iTunes assures me that these trends follow me across services. Top genres of tracks that I added to my iTunes library are:

Electronic (70 tracks)
Pop (30 tracks, thanks Lil Nas X)
Dance (14 tracks)
House (9 tracks)
Alternative (8 tracks)

iTunes tracks can only have 1 genre, rather than the many that tracks can have in Last.fm and Spotify, so this is going to, by nature, be a simpler list.

In addition to top songs and top genres, Spotify also looked at my top artists. I apparently listened to 3699 artists this year:

This number is far higher than the number I have in Last.fm data, 3282 artists! Why is that? Spotify stores every artist that appears on a track as a comma-separated list of artists and is able to count each one of them separately. Last.fm does not. That is why. Because of data storage decisions.

In addition to the total number of artists I listened to, Spotify also tells me what my top 5 artists were of this year.

According to Spotify, my top 5 artists are:

Fred again..
Jacques Greene
Parra for Cuva
Bicep
Smoking Popes

Most of these are artists that I binge-listened to for multiple days, but besides Fred, I’m not sure how many of them really stood up year-round and cross-service.

If I look at my Last.fm data again, which of course covers more than just Spotify, I have the following top 5 artists:

Fred again..
Bicep
Jacques Greene
Parra for Cuva
O’Flynn

So I guess changing up the order and swapping out O’Flynn for Smoking Popes, that list is pretty accurate.

Looking at my top 10 consistently-listened to artists, the story is a bit more interesting. You see that Bicep, Bonobo, Catching Flies, Jayda G, Logic1000, O’Flynn, and Parra for Cuva had the most staying power. The interesting thing about these artists though is that I only listen to a few tracks by each of those artists consistently.

And finally, Spotify took care to identify that my top artist of the year was Fred again.. Indeed, this was the theme across all the services. According to Spotify, I listened to Fred again.. for a total of 1,138 minutes. Let’s dig into what the “real” numbers are across platforms…

The scale of my Fred again.. obsession

I discovered Fred again.. last year, apparently, but it wasn’t until his album dropped in April that I fell into a full-scale obsession. Spotify says I spent 1,138 minutes listening to Fred again.. out of a total of 62,214 minutes. That’s 1.83% of the total time I spent listening to music on Spotify.

If I look across all platforms (that are tracked by Last.fm), and I look at a track-by-track basis, it turns out that of all the tracks I listened to so far this year, 5.71% (5.70% at the time of this writing) of the tracks were by Fred again..

To put that in perspective, here’s a table of my past top artists with their respective percentage shares of my total listens for that year:

Year	Artist	Percentage	Total Listens
2017	The xx	1.86%	8,901
2018	Poolside	1.50%	10,836
2019	Tourist	2.49%	10,394
2020	Tourist	1.37%	16,579
2021	Fred Again..	5.71%	16,504*

* year incomplete

Of course, that’s looking at the number of listens as a share of total tracks. Back in 2019 I declared that based on listening time alone, I’d listened to Tourist for nearly 6% of the total time spent listening. Perhaps Fred has some rivalry after all.

I also want to point out that for some tracks on Soundcloud, I was listening to DJ sets by him that were not properly accounted for in metadata, which is clear when you see the tracks on my Soundcloud 2021 playback

I have another chart where I examine the distribution of track listens for my top artists—what is the average number of listens per track for each artist?

For Fred again.., that number is 13 listens per track. The next closest artist is Jim-E Stack with 9 listens per track, but again, let’s look at scale. 13 listens per track for 72 tracks by Fred again.. (singles are listed separately from album tracks), and 9 listens per track for 16 tracks by Jim E-Stack. It’s the difference between obsessing over a single EP and some back catalog of an artist, and obsessing over the entire available catalog of an artist.

This is a lot of data, and the numbers are clear that I love Fred again..’s music. I wasn’t lucky enough to snag tickets for his LA or NYC shows, but I will 100% see him the next time he makes it stateside. If you want to check him out, head over to Soundcloud and listen to my favorite song of his, Eazi (DoItNow):

The tops of Spotify top 100

Along with Spotify Wrapped, Spotify puts together a playlist of 100 tracks that were most listened to and relevant I guess, to my year. If you want to listen along, you can find my playlist here. (It’s fun to note that the playlist itself isn’t actually shareable directly, I guess because breaking customer expectations of how links work is a “great” way to do marketing.)

Given that not all the songs I listened to and enjoyed were on Spotify this year, my Spotify top 100 playlist is going to be missing some notable favorites beyond Eazi (DoItNow), such as Jacques Greene’s stunning edit of Frank Ocean’s track White Ferrari.

That aside, it turns out that the tracks on the playlist aren’t a 1:1 listing of your top 100 most played tracks. I find this to be fascinating, and makes it clear how much thought Spotify puts into making listenable playlists vs ones that are direct reflections of ~ the data ~. As the Spotify Wrapped FAQ puts it:

Yes, for the most part your playlist is ordered by the songs you played most frequently to least frequently.

“For the most part” is doing a good bit of work there. Let’s dive deeper!

The rankings are based on a combination of the number of plays and the unique number of days you listened to them: The top 5 songs and top 5 artists are based on playcount. The ranking of the top 100 songs playlist is initially sorted by playcount, but after track 10, we apply some artist separation so it’s not a direct reflection of playcount.

Cool! Artist separation, for those playing along at home, is basically there to make sure that if you perhaps listened to Fred again..’s entire discography, your top 40 tracks wouldn’t just be Fred again.. tracks.

Let’s look at some numbers for my playlist. Top 5 tracks are:

track_name	artist	first_discovered	total_listens¹
Out Here with Us	Parra for Cuva	April 11, 2019 14:33:28	15
Ordel	Parra for Cuva	February 10, 2021 11:10:00	11
Both Of Us - Edit	Jayda G	October 16, 2020 16:47:39	13
Dermot (See Yourself in My Eyes)	Fred again..	April 16, 2021 11:47:31	36
Kyle (I Found You)	Fred again..	December 15, 2020 14:35:22	61

Only one of those tracks is new for 2021. Several tracks out of the 100 have only 1 or 2 plays, but they’re tracks by much-loved artists (Bicep, Gilligan Moss), so it wasn’t surprising to see them on the playlist. It feels like a relatively good reflection of my year in music, notwithstanding the lack of the last 2 months of the year.

The playlist of 100 songs represents about 56 total unique artists, 32 of whom I discovered in 2021. I ended up buying at least 57 of those top 100 tracks, making it clear that they were important to me (consumerism is certainly a reflection of value in this context).

My top albums of 2021

I didn’t expect to see Max Richter’s Sleep album on here, but what it turns out happened there is that it’s an 8.5 hour album, and Fred again.. used a couple songs to soundtrack some instagram stories that he posted, so I put on the album and got to work. What ordinarily might have indicated a multi-day or multi-week obsession with an album that was 13-20 tracks long was actually a one or two day jaunt through an album with 31 tracks.

You can see that difference played out when you compare the top 10 albums by number of listens to the top 10 albums by consistency. What’s entertaining to me is that half of the consistently-listened-to albums are full albums, but the other half are singles or EPs.

What were the _best_ albums of 2021 in my opinion? Probably (in an unranked list):

Bicep - Isles
DJ Seinfeld - Mirrors
Fred again.. - Actual Life (April 14 - December 17 2020)
Fred again.. - Actual Life 2 (February 2 - October 15 2021)
Gilligan Moss - Gilligan Moss
Lil Nas X - Montero

Best EPs:

TSHA - OnlyL
Jim E-Stack - EPHEMERA
Logic1000 - You’ve Got the Whole Night to Go

Best single:

salute - Want U There
Caribou - You Can Do It

BPM and its flaws

I asked on Instagram what types of data people were interested in learning more about, and one person asked for BPM data! This is a fun one, honestly, because BPM data is surprisingly hard to come by.

Spotify doesn’t include BPM data in its Spotify Wrapped, although it does include BPM as a “tempo” field in track metadata. Last.fm doesn’t store BPM data, and only tracks that I’ve purchased on Beatport (not iTunes or Bandcamp) include BPM in their metadata.

So I can calculate some limited stats based on the tracks that I’ve purchased in 2021, which is a grand total of 42 tracks. Based on those tracks, the average BPM of what I listen to is quite clearly “house music”. But of course, that’s going to be heavily biased towards dance tracks, because Beatport is a purchasing site of last resort for me, and also pretty much only sells music intended for DJing (it’s tough to find the non-extended version of a track there).

Highlights of discovered artists

I discovered a whopping 1369 artists this year. Fewer than last year, and still a good number of those are random things I watched on YouTube that were incorrectly scrobbled as tracks, like “AI BREAKS NES TETRIS!” or “Climbing with #1 Arctic Elite force” or even “Engineer Explains Every Roller Coaster For Every Thrill”.

Some other artists end up showing up as new discoveries due to how data is stored, such as: “Bleachers, Organized Noize” or “Brittany Howard, Joy Anonymous & Fred again..”. I discussed why this happens in my post about this last year as well.

There’s no easy way for me to clean this data up, and because I’m not really using this data for anything more than my own entertainment, the incentive to invest the time to clean it is very low.

Spotify didn’t share how many artists I discovered this year, but a good number of them were on Spotify, but many of the notable discoveries were facilitated by Spotify (as in, I went to Spotify to listen to them first) but the conduit for discovery was not entirely through Spotify’s algorithms.

Some notable discoveries from my top 20 discovered artists are:

Lil Nas X, whose album Montero was a genre-spanning wonder. My favorite track is probably: Lost in the Citadel, which is a very pop-punk-sounding track.

DJ Seinfeld, who I’d seen live in 2019 as an opener for Jon Hopkins (but was underwhelmed), put out an excellent album Mirrors, which I enjoyed enough to see him live last week. My favorite track by him is These Things Will Come To Be.

Haiku Salut, who showed up on a Discover Weekly playlist with a few catchy tracks. It’s impressive to me how catchy I found it and I ended up buying it that same week, Pattern Thinker.

Moonshine, a DJ collective in Montreal that I discovered while randomly browsing Twitch last year, who put out an album called SMS for Location vol. 4 that I really enjoyed. I recommend their track Woah.

CRi, an Anjunadeep artist who manages to have some very catchy tracks but also leaves me wondering if I’m listening to someone else the whole time. Mysterious. I enjoy this track especially: Friends in Secret.

Everyone You Know, who was another artist hyped by Fred again.. with their collaboration with Joy Anonymous called “Just for the Times”, which is supremely catchy.

Rochelle Jordan, who Tokimonsta recommended on Twitter and whose track Already is a stunner.

Kilder was another random and recent discovery, probably from an algorithmic playlist, and mostly got addicted to his track Don’t Hide.

Sonnee I’m not sure how I encountered, perhaps through a playlist curated by someone. His track So What? has major Tourist vibes.

Some of these artist discoveries translated directly into my concert activities (or resulted from seeing what other shows people were going to, with my discovery of Adrianne Lenker).

My year in concerts

Unlike 2019 and 2018, where I went to nearly a show a week, this year and last year I stayed home most of the time. As things have reopened and venues in San Francisco are required to abide by a vaccination mandate and a mask mandate, I’ve ventured out a few more times this year.

I ended up going to 7 shows and saw a total of 15 artists, some of whom were new to me (Jayda G, Booka Shade, Black Coffee) and others who were familiar (Amtrac, DJ Seinfeld, Jacques Greene).

Aside from going to concerts, I’m listening to music pretty consistently, and moreso than in years past (as evidenced by my “minutes listened” metric.

When am I listening to music?

There are clear trends when I look at WFH years compared to a non-WFH year. When I’m in an office, I listen to less music. Likely because I have the sounds of people to keep me occupied, instead of the sounds of music?

The trend is more clearly relevant to my workdays when you see what time of day I’m listening to music. The starkness of this graph is partially due to the amount of time that a workday is compared with other portions of the day, but the difference is still stark.

What is a workday? Here’s the statement where I define the time of days:

| eval time_of_day=case(in(hour,"10","11","12","13","14","15","16","17"),"probsworkday", in(hour,"18","19","20","21","22"), "4", in(hour,"23","24","00"), "5", in(hour,"01","02","03","04","05","06","07"), "6", in(hour,"08","09"), "1")

| eval time_of_day=case((time_of_day=="probsworkday" AND in(weekday,"1","2","3","4","5")), "2", (time_of_day=="probsworkday" AND in(weekday,"0","6")), "3", time_of_day!="probsworkday", time_of_day)

So essentially, I define a workday as between 10am and 5:59pm on Monday – Friday. This is no longer quite as accurate as it used to be for me, working a new job has me up earlier, but the pattern in the data still holds.

How much money?

It’s important to dig into how much money Spotify pays artists, especially as half the theme of this post is unpacking the veracity of Spotify’s Wrapped data campaign as it compares to my actual behavior (inasmuch as it’s quantified).

If I assume that every song I listened to this year was one that I streamed on Spotify, and if we use the commonly-reported stat that the average payment to an artist from a single Spotify stream is $0.001128, then that rough math means that:

My individual Spotify streams paid artists a total of $18.61.

I paid Spotify $120 this year to access their service.

Again, that’s assuming that every song I listened to this year, I listened to on Spotify.

Of course, I am a diligent music purchaser. I bought a total of 453 songs (so far) this year.

Data caveat: I determine if a song came from Bandcamp by regexing the site URL from the comments of the track. If for some reason a track doesn’t mention that in the comments, well, it’s missing from this data. iTunes data is reliably captured. I also purchase tracks from Beatport, but I have no reliable way of identifying those in the data, and I’m not going to identify those tracks manually, so they’re missing completely from this analysis.

So, let’s make up some more estimates and numbers.

If I assume that every song I bought on Bandcamp cost me $1.50, and I use the quote from the CEO in this Complex interview, 82% of every purchase went to artists. Some music I did purchase on Bandcamp Fridays, which meant more of my money went directly to artists. Some music was cheaper than the $1.50 estimate I’m using here, so given that:

I purchased 327 songs from 104 artists so far this year on Bandcamp.

That produces roughly $402 in profits for the artists.

What I can’t find on Bandcamp, I buy on iTunes (out of convenience, largely). It’s a bit difficult to find consistent stats on what share of an iTunes track purchase makes it to the artist. According to a forum post from 8 years ago, Apple takes 34% of a track cost. The typical iTunes track costs $1.29, except Lil Nas X who made every track on his album $0.69, and the odd $0.99 cent wonder. But let’s take the high estimates! Let’s see how it compares! Given that:

I purchased 126 songs by 30 artists so far this year on iTunes.

That leads to roughly $107 in profits for the artists.

A less charitable number was $0.09 from every track makes it back to the artists. In that case, those artists made $11 from my purchases.

Like I said, most of these numbers are effectively made up. It’s impossible as an individual consumer to know exactly how much of my music consumption activity led to profits for the artists I’m listening to, due to the intricacies of royalties, licensing, and contracts between digital streaming services, labels, artists, producers, songwriters, and more.

It does seem pretty clear, however, that buying music on Bandcamp is always a good idea.

Added commentary

I cover a lot with this data-driven rundown every year, but the thing that remains constant is my love of music. I love listening to music, discovering new artists, recommending music to friends, and going to live shows and DJ sets.

The latest Money 4 Nothing podcast discussed the way that Spotify Wrapped serves as an emblem of the datafication and financialization of culture, especially whether the number of listens can serve as an adequate proxy for the value of music to a person or a culture.

I think that’s worth considering. These numbers reflect what music and which artists are important to me by virtue of their presence in my life, but that can be separated from what is meaningful or culturally relevant.

There are also limitations to the listening data, given that a large portion of it is algorithmically derived, or derived from an aura reader. This is especially clear given an audio aura described as ecstatic and innovative, yet one of my top artists on Spotify was the sad punk band Smoking Popes. (I highly recommend them if you’re in a wallowing mood).

Data makes experiences seem objective, because they’ve been quantified, but that isn’t actually true. All data is a proxy representation, a measurement of something distinct, and is often an imperfect measurement (how do you objectively measure energy? danceability?). Even the definition of a listen differs across services.

Another point that Saxon Baird and Sam Backer make on their podcast is that with the datafication of habits, the subjectivity of self awareness can be frayed. I agree with this, and also recognize that most things affect the personal subjectivity of what I enjoy listening to. Being in a community of music fans affects which artists I’m apt to highlight as my favorites, or even to reinforce my listening habits as I go about my life. Part of that is just how life works these days, which they point out too, drawing a line between radio and record labels dictating taste whereas now we have Spotify and algorithms largely “curating” for us.

It’s easy for me to recognize that Fred again.. was an objectively and subjectively important artist to me this year, but without the data I might not have remembered that Litany’s 4 track EP was on repeat for several days, or the scope of my brief obsessions with Jim-E Stack’s EP EPHEMERA. There is a component of transactive memory to the quantification for me as well, making it easier for me to remember my subjective experiences better if the “objective” data is being stored for me elsewhere.

That’s all for now. I’ll be here again next year with my annual review of what I listened to, and in the meantime, if you want to know anything else or want a music recommendation, you can find me on Twitter!

Total listens according to Last.fm. ↩︎

Top Business / Management / Leadership Books by and/or about Womxn

Fri, 05 Nov 2021 10:21:52 +0000

I’ve been listening to the Farnam Street podcast, The Knowledge Project, recently and enjoying the guests that have talked about The Personal MBA or Relationships vs Transactions. But I noticed a pattern. I realized that the guests were largely telling stories about men, mentioning books by men, and I didn’t see myself in these conversations. When I went to dig deeper into the recommended reading, I found more of the same.

I’m not trying to pick on Farnam Street, but the institutional blindness of having a slogan: “Our Content Helps You Succeed In Work and Life” without examining who is behind that “You” is real. So I dashed off a quick tweet about my frustration, and gosh did the Twitterverse deliver.

https://twitter.com/smorewithface/status/1456141809525035008

You can read the full replies to that tweet if you want to see all the attributed recommendations, but I’ve gathered them here in a loose structure. If you want the unstructured list, check out this published Google doc compilation I created.

Managing and Leading People

Leading an Organization

Founding and Building a Business

Working Better

Work more efficiently or productively

Growing Yourself (At Work, Maybe)

Widen Your Perspective

I also recommend using the Library Extension to automatically search your local library catalog for these books.

Thanks to the recommenders

Many thanks to Better Allies, Kim Moir, David Ryan, Alice MacGillivray, Jillian Kozyra, Margaret Fero, Laura Glu, Liz Wiseman, Linda van der Pal, Sophie Weston, Mariposa Leadership, Richard Hughes-Jones, Katherine Collins, Arie Goldshlager, Michele Zanini, Bob Sutton, James Addison, Davis Liu, MD, Suva Chattopadhyay, Anna-Lisa Leefers, Neil Hodgson, Mindy Howard, leverup, Jo Miller, and Jeff Tetz for recommending these books, and to everyone that retweeted my request as well.

Inc. published a similar list that you likely want to check out as well: 60 Great Business and Leadership Books, All Written by Women. Thanks to Shantha R. Mohan, Ph.D., DTM for the pointer!

One last thought

I asked for recommendations of business, management, and leadership books by and about women, and I got so many more than I expected! As I dug through the list of recommendations, noticed a new pattern—most of the authors look like me, a white cis woman.

Many of the authors have degrees and/or positions at Ivy League universities. Some of these books seem to espouse a kind of “Lean In feminism”, where if you work hard enough in the existing system, or change yourself to work with the system, you’ll succeed. That doesn’t work for everyone, and can even work against people.

There’s an innate bias to who gets published, and it’s worth considering whose voices we might not be listening to in the room, who doesn’t feel comfortable enough to talk in the room, and who isn’t even in the room. (In this case, the room is a list of crowdsourced book recommendations).

Despite publishing this list of book recommendations, you might not need a book.

As Don Jones (yet another dude) interviewed on the Tech Lead Journal podcast put it, “Define what success means to you” and go after it. And bring others up with you.

And while we’re at it, let’s build a new system where everyone is empowered and supported to find their own success—beyond mere survival.

From Nothing to Something with Minimum Viable Documentation

Tue, 21 Sep 2021 15:50:00 +0000

More and more startups and enterprises are recognizing the importance of high quality product documentation, but it’s tough to know where to start. I’ve taken a few enterprise software products from “nothing to something” documentation and this is the framework I’ve built for myself to create MVD—minimum viable documentation.

If you’re a technical writer trying to find your footing, or someone who cares about adding user documentation for your software but have no idea where to start, this is the guide for you.

What is minimum viable documentation?

If documentation is a product (and it is), minimum viable documentation is the bare minimum documentation that is useful and helpful to customers.

Something good is better than something chaotic and unhelpful, and it’s much better than no documentation at all. It’s also easier to focus on getting to minimum viable documentation rather than trying to reach full-featured documentation as soon as possible, because you’re a human with a life that is not your job.

You might be working with a fully-functional software product that has no useful documentation. In that case, getting to full-featured documentation isn’t your primary goal—getting to minimum useful documentation is. So let’s get started.

Define minimum viable documentation for your product

Before you can write MVD, you need to define what it is for your product. MVD differs depending on your market, customer base, product type, your pricing structure, and more.

I recommend you do the following to define what MVD looks like for your product.

1. Talk to your colleagues

Your goal with these conversations is to get a good understanding of who the target user is for your product and the goals they want to accomplish with your product.

If you have product management, start with them. Find out as much as you can about why the product is being built, who it’s for, and how the product is being positioned in the market.

Also talk to engineering management or senior engineering subject matter experts (SMEs). What user problems are the software trying to solve? What level of expertise do the engineers assume the user has?

If you’re lucky enough to have a sales or marketing team, talk to them. Because of their efforts defining the customer journey, they can help you understand who the audience is and what the key success workflows look like. Who is the product targeting? Why do they want to use this product? What problems are they trying to solve?

Talk to the user experience designers to get an understanding of the user personas they’re designing for and what workflows they think have the most friction. You can also get a sense for how the team approaches their role, whether they’re more focused on designing friction-free workflows or pixel-perfect screens.

After talking to PM, EM, UX, and marketing, you can do the following:

Identify what level of expertise a typical product user has, both with the domain and with the product. This functions as your audience definition.
Write down the main goals of a user before and after they start using your product. What motivates the user?
Map out the key workflows that a user is going to perform in the product. What tasks are the user trying to accomplish?

2. Perform a documentation competitor review

It’s always a good idea to know what your competitors are doing! If you’re not sure what products your product is competing with, ask your sales or marketing people for a list or do some research on your own.

Pick 3-5 companies to focus on, such as your strongest competitors in terms of funding, usage, or closeness to what your product does.

You also want to make sure you’re not benchmarking off useless garbage when you perform your competitor review. In addition to the 3-5 competitors you identify, pick a couple industry leaders or companies that your colleagues mention as having good documentation, such as Stripe API docs, Microsoft docs, or even IBM docs, and include one or two of them in your competitor analysis.

The advantage of choosing the documentation of a couple larger products to review is that they tend to have established documentation teams and offerings in a variety of markets. This makes it easier to find a product that is well-documented and at least somewhat adjacent to what your product does.

The goal of your competitor analysis is to identify how the companies provide documentation about their product(s). Pay attention to the following:

How is the documentation structured?
- By feature?
- By use case?
- By persona?
What documentation is provided?
- What workflows are covered?
- What seems left out?
What type of documentation is it?
- Lots of conceptual information about how the product works?
- Heavy reference information but light on the how-to tasks?
Who is the documentation targeting?
- Look for introductory content or tutorials
- Is there advanced developer content?
How is the documentation site built?
- Use the inspect element option in your web browser or a site like https://builtwith.com/ to figure out what technology the documentation site is built with.
Anything else interesting?
- Does a company have an interesting way of differentiating beta functionality?
- Are code samples hidden behind toggle-to-expand options?
- Is there a plethora of gifs, videos, or other multimedia in the documentation?

Document your findings, of course, and feel free to share your findings with your team during a demo. After all, they might be wondering what you’re doing if you haven’t started writing yet.

3. Assess the current state of your documentation

If there is any sort of documentation for your product, you want to know what it is. It might be a sad README and some code comments, it might be detailed multilayered documentation without much organization or clear goals.

To get a sense of the current state, I recommend doing the following:

Audit the existing content. Identify which topics are covered in the documentation already, and where. Make a list, and also keep track of what topics seem to have a lot of detail, and what you suspect might be outdated. This is a cursory audit, not an in-depth one that you might perform if you were migrating content.
Look at the documentation analytics. If you have analytics for the documentation site, take note of which pages are most frequently viewed, which pages might be serving as entry points, and how much time people spend on various pages.
Talk to your team and get their thoughts on the current documentation. Who has been writing it so far? Are they attached to any topics in particular? Do they share specific topics with customers regularly?
Interview customers of the product and documentation to see what they want to see or find most useful today.

Depending on the quality of the existing documentation, these steps might not be that helpful in informing your approach, but help you set benchmarks for documentation growth and quality, plus identify links you likely don’t want to break.

If you don’t have any documentation, still talk to your team and customers. If you can’t talk to customers for some reason, you can look for discussions about the product on social media like Reddit, Twitter, or Hacker News to identify themes that people ask questions about or really enjoy about your product.

A brief note about terminology: As you review competitor and existing documentation and interview internal and external folks, you might find that your product has some inconsistent terminology. At this stage, you might want to delay the writing process while you create a definitive list of terms to use for the product. This type of work can take more time upfront but it’s easier to create consistency from the beginning than to apply it after the fact.

Define the structure of your documentation

Before you start writing, you want to create a structure or a framework to place your topics into.

The structure for your MVD is directly informed by the work you did to define what MVD looks like for your product, plus some information-architecture-specific research.

Revisit your conversations with colleagues. What workflows and functionality might be important to highlight? Who is buying your product? Who is using your product?
Refer to your competitor review notes. How did your competitors and benchmark docs structure their documentation?
Research information architecture best practices. Refer to some key articles from the Nielsen Norman Group, as well as the book How to Make Sense of Any Mess by Abby Covert, and the associated worksheets.

After this research, draft up some chapter headings and possible topic titles to start with, then get feedback from your UX, PM, Engineering, and Sales and Marketing folks. How accurate, relevant, or helpful does the new structure seem? Have you made any assumptions that don’t make sense for the customer base?

Expect this information architecture to change as you write the MVD and especially as you develop full-featured documentation. This is the nature of a minimum viable product! Put a task in your backlog to plan to refine the structure after you finish MVD and are approaching full-featured documentation so that you can iterate without confusing your customers with frequent changes, and plan so that you don’t break any links.

After you design the initial information architecture, you can start writing.

Start writing minimum viable documentation

So you know what minimum viable documentation might look for for your product, but how do you get there? MVD is all about creating useful content for your users, so start with the entry content!

Focus on key information for customers

As with any “minimum viable” approach, you’re trying to get a basic functional framework down before you start improving it. As you lay that framework, be mindful of scope creep.

Think back to the key workflows that you mapped out earlier. Broadly cover the top few workflows and then flesh out details as you get more comfortable with the product and understand the user goals better. Why go broad instead of going deep into a specific workflow? You’re still learning what the customer finds useful, and what level of detail they might want or need about a specific workflow.

If you spend a lot of time writing a highly detailed workflow that you thought was important and it turns out it’s actually pretty intuitive for customers—that’s time that you could have used to write about something that was really confusing and holding back customers from succeeding with your product.

It’s likely that you’ll encounter cases and situations that you want to write more about. That’s great! Write them down and put them in a backlog to address later. For now, you want to stay focused on these minimal workflows to build out the minimum viable documentation for your product. You can get fancy with use cases and in-depth examples later.

Identify the simplest path to success

Within those broad key workflows, start with the simplest path to success, the “happy path” that most of your customers will take.

That might involve writing a series of topics like:

“Get started using Best Product Ever”
“Install Best Product Ever”
“Set up Best Product Ever”
“Accomplish Straightforward Task in Best Product Ever”.

Get those written, reviewed, and published and start helping people use your product that much sooner.

Clarify any complexity

After you write the documentation to support the simple path to success, what do you write next? Documentation that unravels where complexity lurks in your product.

Depending on your product familiarity, you might need to take more time to research and lean on technical subject matter experts (SMEs) a bit more to write this, but it’s worth it. This documentation content might be topics like:

“Configure the Weird Setting You Must Touch”
“All About This Task That Everyone Wants to Do but No One Can Find”.

You don’t want to get bogged down in documenting around product complexity here. Stay focused on the complex aspects of the key customer workflows, and the crucial information customers need. What might confuse someone if you left it out? What assumptions have you been making about the user that need to be made explicit?

This is often the step when I remember to write things like software requirements, role-based restrictions to functionality, or other crucial cases that are often assumed when developers write their own documentation.

Get feedback and iterate

I assume you’re focusing on minimum viable documentation because you have more work than you have time to complete it. That’s why it’s important to iterate. Yes, I just harped on the importance of prioritization and focus—and it’s essential to make sure that what you prioritize and focus on is still important.

Check in with product management and engineering management regularly (I’d recommend weekly for an every-few-months or less release cadence) about what you’re prioritizing and why.

This check-in is mostly about getting signoff and validation, not direction—but don’t ignore the direction that PM and EM can offer you! If there are important releases coming up that will affect one of the key workflows on your list, you might want to document that workflow sooner, or in more detail than you might otherwise for MVD.

Use these conversations as a way of discovering what customers are paying attention to, and what your PM and engineers are paying attention to as well.

As you send your documentation out for technical review, you might also get feedback that you can use to improve your approach to MVD. With any luck, much of the feedback will duplicate what you have planned—and that’s helpful validation for your approach.

You might get so much feedback that you have to dump a lot of ideas into “plans to write this later” and a backlog that feels like it’s spiralling out of control, but if you stay focused on your scope, you’ll get to that backlog sooner and with a more comprehensive understanding of your documentation and your customers.

If the direction and feedback you get from your team is pretty far removed from your approach to MVD, it’s helpful to discuss why that might be and treat it as prioritization guidance for your future plans. Maybe you misunderstood a key target customer, or the purpose of the product in the market. You might discover you need to realign your understanding and vision of the documentation with that of your team.

What’s next after MVD?

When do you know that you’ve reached minimum viable documentation? It’s somewhat of a fuzzy line. When you notice that you’re writing documentation by adding to existing topics, or writing net new example content, or documenting new features instead of existing features — you’ve moved past MVD and into shaping full-featured documentation.

As you start shifting into that mode, you’re no longer focused on creating the skeleton structure to build off of, but filling in the details and settling into the usual work of modern technical writing.

1. Go through the backlog

Start going through your backlog of ideas. Revisit those ideas and group similar ones together, adding audience definitions, acceptance criteria, and learning objectives where you can. Note who the technical SMEs are and whether any upcoming releases are relevant for some of the tasks.

Ideally, you’re storing this backlog in the same spot as your engineering backlog so that your work is visible to the engineering team.

Work with PM or EM to prioritize those tasks and start working through them. As any writer for a fast-paced development team knows as well, the product development often happens faster than you can write about it, so you’ll never run out of tasks in your backlog.

2. Suggest product improvements

As you went through a flurry of documentation writing to produce MVD, you likely identified some parts of the product that might need to be improved. Again, work with your PM and engineering teams to discuss possible product improvements.

You can also suggest product improvements that directly involve the docs, doing a review of UI text in the product, or auditing pages in the product to suggest opportunities for in-app documentation or context-sensitive help. This is a great opportunity to partner with the UX team as well.

Partner with your engineering and UX teams to make suggestions and build those relationships based on your newfound product and customer expertise.

3. Write use cases and examples

To create more useful content for your customers, you probably want to flesh out specific example scenarios for using your product. You might have written some already as quick start use cases for getting started with your product, but you likely want to write more for the next stage of customer product understanding.

You can use example content to describe customization options for the product, or highlight domain-specific use cases for a market that your customer might be trying to break into.

4. Ask for feedback

You put all this effort into creating minimum viable documentation, but how viable is it really?

Ask your technical SMEs, sales and marketing teams, customers, really anyone that might interact with the documentation internally or externally if they have feedback on your documentation improvements and information architecture.

You could perform some tree testing with the MVD structure to see if there are some improvements you can make to the information architecture as you flesh out the documentation, or just have short conversations with stakeholders.

Use the feedback you get to help shape priorities for your backlog. However, don’t treat all feedback you get as tasks that you must perform—if someone asked for it, it must be important, right?

Instead, validate feedback against your target audience definitions and user goals. Sometimes you’ll get feedback relevant only to a specific edge case that doesn’t make sense to document in the official documentation, or feedback related to a product bug that isn’t something necessarily appropriate to address in the documentation.

5. Review analytics

Review documentation site analytics. Analytics are an imperfect source of feedback, but as long as you established a prior benchmark, check to see if the entry-level pages that you created or updated are the most popular pages.

Are the pageviews higher, or at least somewhat proportional to the user base of your product?
Are there any surprising outlier pages that have a lot of views that you might want to focus on?
What search terms are popular?

You can use these to inform your plans and priorities.

Get from nothing to something with MVD

It can be intimidating to create a set of documentation for a product from scratch, but I hope this post outlines a basic approach that can help.

Start by defining what MVD looks like for your product by talking to colleagues, performing a competitor review, and assessing the current state of documentation. Then do some additional research and define the initial structure of your documentation.

After you’ve laid the groundwork, start writing. Focus on key information for customers and identify the simplest path to success. Clarify any product and task complexity, and seek out feedback. Regularly make changes to what you’ve written as you learn more about the product and your customers.

As you evolve beyond MVD to full-featured documentation, work through your backlog, suggest product improvements, write use cases and examples, and continue asking for feedback. You can also review site analytics to get a sense of how far you’ve come and what you might want to focus on next.

Whether you’re a professional technical writer, a committed startup founder, a generous open source contributor, or someone else, I hope you can use this framework to document your software product.

I tried my best to create a minimum viable blog post to describe this minimum viable documentation framework. As such, I might not have gone into much depth about how to perform a competitor review, get buy-in for terminology proposals, or how to handle the full range of feedback you might receive on your documentation.

If you have feedback or questions for me, or want to see more details about a specific topic, don’t hesitate to reach out on Twitter @smorewithface.

How can I get better at writing?

Wed, 23 Dec 2020 07:41:57 +0000

As a professional writer, I frequently get asked, “as a ______, how can I get better at writing?” I’ve never had a good list of resources to point people to, so I finally decided to write one. I’ve worked hard to become a good writer, and I’ve had the privilege of many good teachers along the way.

If you’re not really sure why your writing isn’t as good as you want it to be, that’s okay. In this blog post, I’ve identified the strategies that I use to write well. I hope they’re useful to you.

Where to start

Read and write more frequently. You can’t get better without good examples or practice. If you want to get better at writing you need to read more and you need to write more.

Identify what you’re trying to improve. Maybe you struggle with grammar, or in clearly communicating your ideas. Maybe it takes too many words for you to get your point across, or you can’t quite connect with the people reading your writing.

Write accurate content by improving your grammar and word choice

Use a tool like Grammarly, or enable grammar checking in whatever tool you use to write, if it’s available. If you don’t want a mysterious AI reading your writing, you can use other resources to improve specific aspects of your grammar.

Some key concepts to focus on:

Understand verb and noun preposition pairs. See Kenneth Beare’s guide Verb + Preposition Combinations on ThoughtCo., or this Preposition Combinations with Adjectives, Nouns, and Verbs PDF from Washtenaw Community College, including a worksheet for practice.
Understanding complementary verb and noun forms, to help avoid nominalizations. When you use the noun form of a word, you can easily obscure your meaning. See How to Improve Your Writing: Avoid Nominalizations on Wordvice.
Review easily-confused words, often homonyms, to identify words that you might be consistently misusing. This Homonyms Main List from an English learning Russian website is fairly exhaustive.
Look words up in the dictionary. If you’re not sure you’ve used a word correctly, or used the correct spelling of a word, look it up. I do this almost every day. I’m partial to the Merriam-Webster dictionary.

I still struggle with the following (more pedantic) grammar rules:

When do I need to use a hyphen to connect two words? See Hyphen Use, on the Purdue Online Writing Lab website.
Did I split an infinitive? What is a split infinitive, anyway? See Infinitives, on the Purdue Online Writing Lab website.
Does my relative pronoun actually clearly refer to something or do I have a vague “that” or “it”? See Pronouns in the Splunk Style Guide.

The somewhat silly yet practical book, Curious Case of the Misplaced Modifier by Bonnie Trenga, might also be a useful read.

Write helpful content by defining outcomes before you start

Before you start writing something, whether it’s a slide deck, an engineering-requirements document, an email, or a blog post like this one, consider what you want someone to do after reading what you wrote.

Often called learning objectives or learning outcomes in instructional design, defining outcomes can help you write something useful and focused. Sometimes when you’re writing something, other extraneous ideas come to mind. They can be valuable ideas, but if they distract from your defined outcomes, you might want to remove them from your main content.

Some example outcomes are:

After reading this blog post, you can confidently draft a clear document with defined outcomes.
After reading this engineering requirements document, my colleague can provide accurate and helpful architecture feedback on the design.
After reading the release notes, I can convince my boss that the new features are worth an immediate upgrade.

I also want to note that if you write an outcome focused on someone understanding something, rewrite it. It’s tough to measure understanding. It’s easier to measure action. For that reason, I try to write outcomes with action-oriented verbs. For more about writing good learning objectives, see the Learning Objectives chapter in The Product is Docs.

Write focused content by identifying your audience

Who will be reading your writing? What do they know? Who are they? What assumptions can you make about them?

If you can’t answer these questions about the people reading your writing, you won’t be able to clearly communicate your ideas to them. You don’t have to be able to answer these questions with 100% certainty, but make the attempt.

If you recognize that you’re writing something for multiple audiences, consider breaking up the content into specific sections for each audience. For example, architects might care about different content than a UI engineer, a product manager might care about different details than the backend engineer.

If you identify the different needs of your varying audiences, you can write more consistently for each specific audience, rather than trying to address all of them all the time. For more on identifying your audience, see the Audience chapter of The Product is Docs.

Write findable content by considering how people get to it

How people get to your content can influence how you write it. If people use search, an intranet, or direct links to find your content, you might make different decisions about how to structure it.

I always assume that people are finding my content by searching the web. They’ve typed a specific search query, found my content as a result, and open it with the hopes that it is the right content for them.

Consider what people are searching for that can be answered by your content, and write a title accordingly. Spend time on the first few sentences of your content to make sure that they further clarify what your content addresses.

For example, I titled this blog post “How can I get better at writing?” because I expect that’s what a lot of people might type into their preferred search engine out of desperation. I could call it “7 quick tips to improve your writing”, but that’s not how most people type search queries (in my opinion).

Mark Baker’s book, Every Page is Page One, covers a lot of information related to this concept. He coins the term “information scent” to describe the signals that indicate to a person that they’ve found the right content to answer their question, and “information foraging” to describe the process of looking for the right information.

Write readable content by considering the structure

People aren’t excited to read technical content or technical documentation. No one rejoices when they get an email. I get paid to write technical documentation and I still avoid reading it if I can. Because people don’t want to read your content, structure it intentionally.

Write for skimming. Bullet points are often better than paragraphs. Tables are often better than paragraphs.

Put information where it needs to be. If you’re writing a series of steps, make sure the steps are actually in the right order. For example, if something needs to be done before all the steps can succeed, put it before the set of steps as a prerequisite.

You also want to consider the desired outcomes of your content and your audience when you structure your content. It can make sense to focus on one audience in one piece of content, or one desired outcome in one piece of content. Don’t try to do too much in one piece of writing.

Nielsen Norman Group has an incredible set of research and recommendations about how people read and how you can structure your content. I recommend the following articles:

Write clear content by intentionally choosing your words

You want to make your content easy to find and easy to understand. To do this, you need to be consistent and intentional about the words that you use.

Use consistent terminology. This isn’t the time to write beautiful prose that uses different words to mean the same thing. Don’t overload terms by using the same term for multiple things, and don’t use multiple terms to refer to one thing. Use the same terms and use them consistently.

If something is a JSON object, call it that. Don’t call it a JSON object sometimes, a JSON setting other times, or a JSON blob other times. Pick one term and use it consistently. You might have to pick an imperfect term and live with it. It happens! There are only so many words to choose from.

Be intentional about the words you use. Consider the words that your readers use to describe what you’re writing about, and use the same words if you can. Even if those words don’t match up completely with the feature names in use by your product.

If all of your software’s users refer to “dark mode” instead of “dark theme”, you might need to use both terms in your content so that people can find it. For some internal documentation, you might need to make a mapping of internal names that people use for something with the external names used in the product.

If you’re not sure what term to use, find out what terms your readers are already using. If you have access to search query logs of your website search, review those for patterns. If you don’t already have readers or users for your product, you can do some competitive analysis to understand what terms are in common usage in the market.

You can also check the dictionary or use a tool like Google Books Ngram Viewer or Google Trends to identify common terms for what you’re attempting to describe.

Nielsen Norman Group again has some excellent resources on clear writing:

Write trustworthy content by thinking about the future

Errors in content, especially technical documentation, lead to mistrust. When you write a piece of content, consider the future of the content.

The future of the content depends on the purpose and type of content that you’re writing. This list contains some common expectations that readers might have about various content types:

A blog post has a date stamp and isn’t kept continually updated.
Technical documentation always matches the product version that it references.
Architecture documents reflect the current state of the microservice architecture.
An email gets the point across and can’t be edited after you send it.

You must consider the future and maintenance of any content that you write if your readers expect it to be kept up-to-date. To figure out how difficult maintaining your content will be, you can ask yourself these questions:

How frequently does the thing I’m writing about change?
How reliable does my content need to be?
How quickly does my content need to be accurate (e.g., after a product release)?

By answering these questions, you can then make decisions about how you write your content.

What level of detail will you include in your content?
Will you focus your efforts on accuracy, speed, or content coverage?
Do you want to include high-fidelity screenshots, gifs, or complex diagrams?
Do you want to automate any part of your content creation?
Who will review your content? How quickly and thoroughly will they review it?

For more on maintaining content and making decisions about your documentation, see the Documentation Decisions chapter in the book The Product is Docs (which I contributed to).

Feel empowered to write better content

I hope that after reading this blog post you feel empowered to write more accurate, helpful, focused, findable, readable, clear, trustworthy content. This is an overview of strategies. If you want to dig deeper into a specific way to improve your writing, check out the books and articles linked throughout this post.

If you have something you think I missed, you can find me on Twitter @smorewithface.

Wrapping up 2020: Spotify, SoundCloud, and Last.fm data

Thu, 03 Dec 2020 16:25:00 +0000

Another year, another Spotify Wrapped campaign, another effort to analyze the music data that I collect and compare it to what Spotify produces. This year I have last.fm listening habit data, concert attendance and ticket purchase data, livestream view activity data, my SoundCloud 2020 Playback playlist, and the tracks on my Spotify top 100 songs of 2020 playlist.

It’s always important to point out that the data covered in the Spotify Wrapped campaign only covers the time period from January 1st, 2020 to October 31st, 2020. I discuss the effects of this misleading time period in Communicate the data: How missing data biases data-driven decisions. Of course, writing this post on December 2nd, nearly the entire month of December is missing from my own analyses. I’ll follow up (on Twitter) about any data insights that change over the next few weeks.

Top Artists of the Year

Spotify says my Top 5 artists of the year are:

Disclosure
Lane 8
Kidnap
Tourist
Amtrac

My own data shows some slight permutations.

My top 5 artists are nearly the same, but much more influenced by music that I’ve purchased. The overall list instead looks like:

Tourist
Amtrac
Booka Shade
Jacques Greene
Lane 8

For the second year in a row, Tourist is my top artist! Kidnap still makes it into the top 10, as my 7th most-listened-to artist so far of 2020.

Disclosure, somewhat hilariously, doesn’t even break the top 10 artists if I am relying on Last.fm data instead of only Spotify. What’s going on there? Turns out Disclosure is my 11th-most-listened to artist, with 97 total listens so far this year. If I dig a little bit deeper, looking at the song Know Your Worth which Spotify says I’ve listened to the most in 2020 by Disclosure, I can see exactly why this is happening.

Disclosure’s latest album, ENERGY, includes a number of collaborations. Disclosure is the main artist for most of these tracks, but in some cases (like with Know Your Worth, which came out as a single February 4, 2020) the artist can be inconsistently stored by different services.

As a result, the Last.fm data has a number of different entries for the same track, with differently-listed artists for each one. Last.fm stores only one artist per track, whereas Spotify stores an array of artists for each track. This data structure decision means that Disclosure should have had about 127 total listens, and been my 7th-most-listened-to artist of 2020, instead of 11th.

This truncated screenshot shows some examples of the permutations of data that exist in my Last.fm data collection, with a total listen count of 127 for Disclosure during 2020.

I had a sneaking suspicion that my Booka Shade listening habits are primarily concentrated on a few songs from an EP that he put out this year, so I dug into how many tracks my total listens for the year were spread across.

Instead, it turns out that my listens to Booka Shade are actually the most distributed across tracks of all of my top 10 artists. Sjowgren is also an outlier here, because they’ve never released an album, so they only have 15 songs in their overall discography yet still made the top 10 artist listens.

Returning my comparison between Spotify and Last.fm data, Amtrac and Lane 8 are in both top 5 lists. This is somewhat expected, because if I look at the top 10 list for artists that I’ve most consistently listened to—artists that I’ve listened to at least once in each month of 2020—both Amtrac and Lane 8 place high in that list.

Given that only 2 days of December have happened as I write this, it’s unsurprising that I’ve only listened to one artist in every month of 2020.

Top Songs of 2020

Enough about the artists—what about the songs?

According to Spotify, my Top 5 songs of the year are:

Apricots by Bicep
Atlas by Bicep
Idontknow by Jamie xx
Cappadocia by Ben Böhmer feat. Romain Garcia
Know Your Worth by Disclosure feat. Khalid

That pretty closely matches my top 5 list according to Last.fm, with some notable exceptions.

My top 5 tracks according to Last.fm are:

Apricots by Bicep (38 listens)
Atlas by Bicep (32 listens)
Idontknow by Jamie xx (22 listens)
White Ferrari (Greene Edit) by Jacques Greene (21 listens)
That Home Extended by The Cinematic Orchestra (20 listens)

The first 3 tracks match, though of course Spotify has an incomplete representation of those listens—I have 29 streams of Apricots according to Spotify.

However, since I bought the track almost as soon as it came out, I also have another 9 listens that have happened off of Spotify. There were also some mysterious things happening with Spotify and Last.fm connections around that time as well, so it’s possible some listens are missing beyond these numbers.

What’s up with the 4th track on the list, though? Where is that in Spotify’s data? It’s actually a bootleg remix of the Frank Ocean song White Ferrari that Jacques Greene shared on SoundCloud and as a free download earlier this year, so it isn’t anywhere on Spotify. It did, however, make it onto my top tracks of 2020 on SoundCloud:

And again, this is a spot where metadata intrudes again and leads to some inconsistent counts. If I look at all the permutations of White Ferrari and Jacques Greene in my data for 2020, the total number of listens should actually be a bit higher, at 23 total listens:

This would actually make it my 3rd-most popular song of 2020 so far, and I’m listening to it as I write this paragraph, so let’s go ahead and call that total number 24 listens.

The 5th-most popular song and 7th-most popular song of 2020 make the case that I haven’t been sleeping very well this year (though I recall these tracks also showed up in 2019 as well…), because those 2 tracks comprise my “Insomnia” playlist that I use to help me fall asleep on nights when I’ve been, perhaps, staying up too late doing data analysis like this.

You can see the influence of consistent listening habits with top artist behaviors when you look at the top 10 songs that I’ve consistently listened to throughout 2020, with 2 songs by Kidnap, one by Bicep, and another by Amtrac.

To me, though, this table mostly underscores how much music discovery this year involved. I didn’t return to the same songs month after month during 2020. Likely as a result of all the DJ sets I’ve been streaming (as I mentioned in my post about Listening to Music while Sheltering in Place) this has been quite a year for music discovery, and breadth of listening habits.

My top 10 songs of 2020 had a total of 222 listens across them. However, I have a total of 14,336 listens for the entire year, spread across 8,118 unique songs in total.

Even with possible metadata issues, that’s still quite the distribution of behavior. Let’s dig a bit deeper into artist discovery this year.

Artist Discovery in 2020

In my post earlier this year about my listening behavior while sheltering in place, I discovered that my artist discovery numbers in 2020 seemed to be way up compared with 2018 and 2019, but weren’t actually that far off from 2017 numbers.

What I see when comparing my 2020 artist discovery statistics from my Last.fm data and my Spotify data is even more interesting. In contrast to what seemed to be true in last year’s post, Wrapping up the year and the decade in music: Spotify vs my data (For what it’s worth, last year’s number should have been 1074, instead of 2857 artists discovered—data analysis is difficult), Spotify’s data is much higher than the number I calculated this year.

According to Spotify, I discovered 2,051 new artists, whereas my Last.fm data claims that I only discovered 1,497 artists this year.

Similarly, Spotify claims that I listened to 4,179 artists this year, whereas my Last.fm data indicates that I listened to 3,715 artists.

Again, this comes down to data structures and how the artist metadata is stored for each service. I wrote about the importance of quality metadata for digital streaming providers earlier this year in Why the quality of audio analysis metadatasets matters for music, but it’s also apparent that the data structures for those metadatasets are just as important for crafting data insights of varying value.

Because Spotify stores all artists that contributed to a track as an array, I can listen to a track with 4 contributing artists on it, 1 of which I’ve listened to before, and according to Spotify, I’ve now discovered 3 artists and listened to 4, whereas according to Last.fm, I’ll have either listened to 1 artist that I’ve already heard before, or a new artist, possibly called “Luciano & David Morales".

Spotify would store the second artist as Luciano, David Morales, thus allowing a more accurate count of listens for the Luciano artist. Similarly, my artist discovery data includes some flawed data, such as YouTube videos that got incorrectly recorded.

The Billy Joel and Jimmy Fallon duet of The Lion Sleeps Tonight never gets old, but it appears the original video is no longer on YouTube so I’m not going to link it.

This becomes clear in my top 20 artist discoveries of 2020 chart, where BTS and Big Hit Labels are listed separately, although they are both indicative of one of my best friends joining BTS ARMY this year and sharing her enthusiasm with me.

Ultimately I’m grateful that the top 20 artists of 2020 are all artists that I discovered during the pandemic and have excellent songs that I love and continue to listen to. Many of the sparklines that represent my listening activity for these artists throughout the year have spikes, but mostly my listening patterns indicate that I’ve been returning to these artists and their songs multiple times after first discovery. Some notable favorites on this list are KC Lights’ track Girl and Dennis Cruz’s track El Sueño, plus the entire Fennec album Free Us Of This Feeling.

Genre Discovery in 2020

The most-commented-on data insight from #wrapped2020 is probably the genre discovery slide.

According to Spotify, I listened to 801 genres this year, including 294 new ones. I’m not even sure I could name 30 genres, let alone 300 or 800. Where are these numbers coming from?

It turns out that, much like storing artist data as an array for each song, Spotify stores genre data as an array for each artist. This means that each artist can be assigned multiple genres, thus successfully inflating the number of genres that you’ve listened to in 2020.

For example, if I use Spotify’s API developer console to retrieve the artist information for Tourist, with a Spotify ID of 2ABBMkcUeM9hdpimo86mo6, it turns out that he has 6 total genres associated with him in Spotify’s database: chillwave, electronica, indie soul, shimmer pop, tropical house, and vapor soul.

I could start discussing the possible meaningless of genres as a descriptive tool, the lack of validation possible for such a signifier, the lack of clarity about how these genres were defined and also assigned to specific artists, but that’s best for another blog post.

Instead, let’s look at what little genre data I do have available to me more generally.

According to Spotify, my top genres were:

House
Electronica
Pop
Afro House
Organic House

All of these make sense to me, except for Organic House, because I don’t know what makes house music organic, unless it’s also grass-fed, locally-sourced, and free range. Perhaps Blond:ish is organic house.

I don’t have any genre data from Last.fm, since the service only stores user-defined tags for each artist, and those are not included in the data that I collect from Last.fm today. Instead, I have the genres assigned by iTunes for the tracks that I’ve purchased from the iTunes store.

The top 8 genres of music that I added to my iTunes library in 2020 by purchasing tracks from the iTunes store are:

Dance (124 songs)
Electronic (121 songs)
House (78 songs)
Pop (37 songs)
Alternative (27 songs)
Electronica (12 songs)
Deep House (10 songs)
Melodic House & Techno (9 songs)

Clearly, this is a very selective sample, and is only tied to select purchasing habits, which are roughly correlated to my listening habits.

I shared all of this genre data to essentially look at it and go “wow, that wasn’t very insightful at all”. Let’s move on.

Time Spent Listening to Music in 2020

The last metric I want to unpack from Spotify’s #wrapped2020 campaign is the minutes listened data insight. According to Spotify, I spent 59,038 minutes listening to music this year.

According to my own calculations, I spent roughly 81,134 minutes listening to music in 2020.

Let’s talk about how both of these metrics are super flawed!

Spotify counts a song as streamed after you listen to it for more than 30 seconds (per their Spotfiy for Artists FAQ), so it’s logical to assume that this minutes listened metric likely from a calculation of “number of streams for a track” x “length of track” and then rounded and converted to minutes. It could even result from an different type of calculation, “number of total streams” x “average length of track in Spotify library”, but I have no way of knowing if either of these are accurate besides tweeting at Spotify and hoping they’ll pay attention to me.

Unfortunately for all of us, but mostly me, my own minutes listened metric is just as lazily calculated. I don’t have track length data for all the tracks that I listen to and I don’t know at what point Last.fm counts a track as being worthy of a scrobble. I do have a list of how much time I spent listening to livestreamed DJ sets online, and I do have some excellent estimation skills. I calculated my number of 81,134 minutes so far in 2020 by calculating and assuming the following:

An average track length of 4 minutes
An average concert length of 3 hours
An average DJ set length of 4 hours
An average festival length of 8 hours

Using those averages and estimates, I calculated the total amount of time I spent listening to music across Last.fm listening habits, concerts and DJ sets attended (no festivals this year), and livestreams that I watched online, thus arriving at 81,134 minutes. That doesn’t count any DJ sets that I listened to on SoundCloud, and certainly the combination of a 4 minute track length estimate with the uncertainty of what qualifies a track as being scrobbled makes this data insight somewhat meaningless.

Regardless, let’s compare this estimated time spent listening in minutes against the total number of minutes in a year.

Beautiful. I still remembered to sleep this year. No matter which dataset I use, however, it’s clear that I’ve listened to more music in 2020 than in 2019. Spotify’s metric for this same time period in 2019 was 35,496 minutes. The less-flawed but less-complete metric I used last year, calculated using the track length stored in iTunes multiplied by the number of listens for that track, indicated that I spent 14,296 minutes listening to music in 2019.

As one final Spotify examination, let’s dig into the Spotify Top 100 playlist.

Top 100 Songs of 2020 Playlist

Alongside the fancy graphics and data insights in the #wrapped2020 campaign, Spotify also creates a 100 song playlist, likely (but not definitively) the top 100 songs of the time period between January 1st, 2020 and October 31st, 2020.

I found my playlist this year to be relatively accurate, perhaps because I spent more time listening to Spotify than I might have in previous years, or perhaps they made some internal data improvements, or both! I often spend more time listening to SoundCloud if I’m traveling a lot, listening to offline DJ sets on plane flights; or listening to Apple Music on my iPhone, with songs that I’ve added from my iTunes library. Without much time spent commuting or traveling this year, it’s likely that my listening habits remained fairly consolidated.

Similarly to what I discovered about my top 10 tracks, I had relatively distributed music interests this year. The 811 total listens for all 100 songs in my Spotify playlist represent just 0.06% of my total listens in 2020 so far.

Despite my overall listening habits being relatively distributed across lots of artists and songs, the Top Songs playlist is somewhat more consolidated, with 69 artists performing the 100 songs on the playlist. Nice.

It’s clear that I spent most of this year exploring and discovering new artists, given that 83 of my top songs of 2020 according to Spotify were songs that I discovered in 2020.

Thanks for coming on this journey through my music data with me. I’ll be back at the actual end of the year to dive deeper into my top 10 artists of the year, top 10 consistent artists of the year, my music purchasing activity, as well as some more livestream and concert statistics to round out my 2020 year in music.

Define the question: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:37:15 +0000

This is the eighth and final post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

In this post, I’ll cover the following:

Define the question you want to answer for your data analysis process
How does data go missing when you’re defining your question?
What can you do about missing data when defining your question?

This post also concludes this blog post series about missing data, featuring specific actions you can take to reduce bias resulting from missing data in your end-to-end data-driven decision-making process.

Define the question

When you start a data analysis process, you always want to start by deciding what questions you want to answer. Before you make a decision, you need to decide what you want to know.

If you start with the data instead of with a question, you’re sure to be missing data that could help you make a decision, because you’re starting with what you have instead of what you want to know.

Start by carefully defining what you want to know, and then determine what data you need to answer that question. What aggregations and analyses might you perform, and what tools do you need access to in order to perform your analysis?

If you’re not sure how to answer the question, or what questions to ask to make the decisions that you want to make, you can explore best practices guidance and talk to experts in your field. For example, I gave a presentation about how to define questions when trying to prioritize documentation using data (watch on YouTube). If you are trying to monitor and make decisions about software that you’re hosting and managing, you can dig into the RED method for infrastructure monitoring or the USE method.

It’s also crucial to consider whether you can answer that question adequately, safely, and ethically with the data you have access to.

How does data go missing?

Data can go missing at this stage if it isn’t there at all—if the data you need to answer a question does not exist. There’s also the possibility that the data you want to use to answer a question is incomplete, or you have some, but not all of the data that you need to answer the question.

It’s also possible that the data exists, but you can’t have it—you either don’t have it, or you aren’t permitted to use the data that has already been collected to answer your particular question.

It’s also possible that the data that you do have is not accurate, in which case the data might exist to help answer your question, but it’s unusable, so it’s effectively missing. Perhaps the data is outdated, or the way it was collected means you can’t trust it.

Depending on who funded the data collection, who performed the data collection, and when and why it was performed, can tell you a lot about whether or not you can use a dataset to answer your particular set of questions.

For example, if you are trying to answer the question “What is the effect of slavery on the United States?”, you could review economic reports, the records from plantations about how humans were bought and sold, and stop there. But you might be better off considering who created those datasets, who is missing from those datasets, and whether or not those datasets are useful to answer your question, and which datasets might be missing entirely because they were never created, or what records did exist were destroyed. You might also want to consider whether or not it’s ethical to use data to answer specific questions about the lived experiences of people.

Or, for another grim example, if you want to understand how American attitudes towards Muslims changed after 9/11, you could (if you’re Paul Krugman) look at hate crime data and stop there. Or, as Jameel Jaffer points out in a Twitter thread, you could consider whether or not hate crime data is enough to represent the experience of Muslims after 9/11, considering that “most of the “anti-Muslim sentiment and violence” was *officially sanctioned*” and therefore, all of that is missing from an analysis that focuses solely on hate crime data. Jaffer continues by pointing out that,

“For example, hundreds of Muslim men were rounded up in New York and New Jersey in the weeks after 9/11. They were imprisoned without charge and often subject to abuse in custody because of their religion. None of this would register in any hate crimes database.”

Data can also go missing if the dataset that you choose to use to answer your question is incomplete.

Incomplete dataset by relying only on digitized archival films

As Rick Prelinger laments in a tweet—if part of a dataset is digitized, often that portion of the dataset is used for data analysis (or research, as the case may be), with the non-digitized portion ignored entirely.

For example, if I wanted to answer the question “What are common themes in American television advertising in the 1950s”? I might turn to the Prelinger Archives, because they make so much digitized archival film footage available. But just because it’s easily accessible doesn’t make it complete. Just because it’s there doesn’t make it the best dataset to answer your question.

It’s possible that the Prelinger Archives don’t have enough film footage for me to answer such a broad question. In this case, I can supplement the dataset available to me with information that is harder to find, such as by tracking down those non-digitized films. I can also choose to refine my question to focus on a specific type of film, year, or advertising agency that is more comprehensively featured in the archive, narrowing the scope of my analysis to focus on the data that I have available. I could even choose a different dataset entirely, if I find one that more comprehensively and accurately answers my question.

Possibly the most common way that data can go missing when trying to answer a question is that the data you have, or even all of the data available to you, doesn’t accurately proxy what you want to know.

Inaccurate proxy to answer a question leads to missing data

If you identify data points that inaccurately proxy the question that you’re trying to answer, you can end up with missing data. For example, if you want to answer the question, “How did residents of New York City behave before, during, and after Hurricane Sandy?”, you might look at geotagged social media posts.

Kate Crawford discusses a study Nir Grinberg, Mor Naaman, Blake Shaw, and Gilad Lotan, Extracting Diurnal Patterns of Real World Activity from Social Media, in the context of this question in her excellent 2013 article for Harvard Business Review, The Hidden Biases in Big Data.

As she puts it,

“consider the Twitter data generated by Hurricane Sandy, more than 20 million tweets between October 27 and November 1. A fascinating study combining Sandy-related Twitter and Foursquare data produced some expected findings (grocery shopping peaks the night before the storm) and some surprising ones (nightlife picked up the day after — presumably when cabin fever strikes). But these data don’t represent the whole picture.”

Because the users of social media, especially those that use Twitter and Foursquare and share location data with those tools, only represent a specific slice of the population affected by Hurricane Sandy. And that specific slice is not a representative or comprehensive slice of New York City residents. Indeed, as Crawford makes very clear:

“there was much more going on outside the privileged, urban experience of Sandy that Twitter data failed to convey, especially in aggregate.”

The dataset of geotagged social media posts only represents some residents of New York City, and not in a representative way, so it’s an inaccurate proxy for the experience of all New York City residents. This means data is missing from the question stage of the data analysis step. You want to answer a question about the experience of all New York City residents, but you only have data about the experience of New York City residents that shared geotagged posts on social media during a specific period of time.

The risk is clear—if you don’t identify the gaps in this dataset, you might draw false conclusions. Crawford is careful to point this out clearly, identifying that:

“The greatest number of tweets about Sandy came from Manhattan. This makes sense given the city’s high level of smartphone ownership and Twitter use, but it creates the illusion that Manhattan was the hub of the disaster.”

When you identify the gaps in the dataset, you can understand what limitations exist in the dataset, and thus how you might draw false and biased conclusions. You can also identify new datasets to examine or groups to interview to gather additional data to identify the root cause of the missing data (as discussed in my post on data gaps in data collection).

The gaps in who is using Twitter, and who is choosing to use Twitter during a natural disaster, are one way that Twitter data can inaccurately proxy a population that you want to research and thus cause data to go missing. Another way that it can cause data to go missing is by inaccurately representing human behavior in general because interactions with the platform itself are not neutral.

As Angela Xiao Wu points out in her blog post, How Not to Know Ourselves, based on a research paper she wrote with Harsh Taneja:

“platform log data are not “unobtrusive” recordings of human behavior out in the wild. Rather, their measurement conditions determine that they are accounts of putative user activity — “putative” in a sense that platforms are often incentivized to keep bots and other fake accounts around, because, from their standpoint, it’s always a numbers game with investors, marketers, and the actual, oft-insecure users.”

Put another way, you can’t interpret social media interactions as neutral reflections of user behavior due to the mechanisms a social media platform uses to encourage user activity. The authors also point out that it’s difficult to identify if social media interactions reflect the behavior of real people at all, given the number of bot and fake accounts that proliferate on such sites.

Using a dataset that inaccurately proxies the question that you’re trying to answer is just one way for data to go missing at this stage. What can you do to prevent data from going missing as you’re devising the questions you want to ask of the data?

What can you do about missing data?

Most importantly, redefine your questions so that you can use data to answer them! If you refine the questions that you’re trying to ask into something that can be quantified, it’s easier to ask the question and get a valid, unbiased, data-driven result.

Rather than try to understand the experience of all residents of New York City before, during, and after Hurricane Sandy, you can constrain your efforts to understand how social media use was affected by Hurricane Sandy, or how users that share their locations on social media altered their behavior before, during, and after the hurricane.

As another example, you might shift from trying to understand “How useful is my documentation?” to instead asking a question that is based on the data that you have: “How many people view my content?”. You can also try making a broad question more specific. Instead of asking “Is our website accessible?”, instead ask, “Does our website meet the AA standard of web content accessibility guidelines?”

Douglas Hubbard’s book, How to Measure Anything, provides excellent guidance about how to refine and devise a question that you can use data analysis to answer. He also makes the crucial point that sometimes it’s not worth it to use data to answer a question. If you are fairly certain that you already know the answer to a question, and the amount of effort it would take to perform data analysis (let alone perform it well) will take a lot of time and resources, it’s perhaps not worth attempting to answer the question with data at all!

You can also choose to use a different data source. If the data that you have access to in order to answer your question is incomplete, inadequate, inaccurate, or otherwise missing data, choose a different data source. This might lead you to change your dataset choice from readily-available digitized content to microfiche research at a library across the globe in order to perform a more complete and accurate data analysis.

And of course, if a different data source doesn’t exist, you can create a new data source with the information you need. Collaborate with stakeholders within your organization, make a business case to a third-party system that you want to gather data from, use freedom of information act (FOIA) requests to gather data that exists but is not easily-accessible to create a dataset.

I also want to take care to acknowledge that choosing to use or create a different dataset can often require immense privilege—monetary privilege to fund added data collection, a trip across the globe, or a more complex survey methodology; privilege of access, to have access to others doing similar research and are willing to share data with you; and privilege of time to perform the added data collection and analysis that might be necessary to prevent missing data.

If the data exists but you don’t have permission to use it, you might devise a research plan to request access to sensitive data, or work to gain the consent of those in the dataset that you want to use to allow you to use the data to answer the question that you want to answer. This is another case where communicating the use case of the data can help you gather it—if you share the questions that you’re trying to answer with the people that you’re trying to collect data from, they may be more inclined to share it with you.

Take action to reduce bias in your data-driven decisions from missing data

If you’re a data decision-maker, you want to take these steps to take action:

Define the questions being answered with data.
Identify missing data in the analysis process.
Ask questions of the data analysis before making decisions.

If you carefully define the questions guiding the data analysis process, clearly communicating your use cases to the data analysts that you’re working with, you can prevent data from going missing at the very start.

Work with your teams and identify where data might go missing in the analysis process, and do what you can to address a leaky analysis pipeline.

Finally, ask questions of the data analysis results before making decisions. Dig deeper into what is communicated to you, seek to understand what might be missing from the reports, visualizations, and analysis results being presented, and whether or not that missing data is relevant to your decision.

If you work with data as a data analyst, engineer, admin, or communicator, you can take these steps to take action:

Steward and normalize data.
Analyze data at multiple levels of aggregation and time spans.
Add context to reports and communicate missing data.

Responsibly steward data as you collect and manage it, and normalize it when you prepare it for analysis to make it easier to use.

If you analyze data at multiple levels of aggregation and time spans, you can determine which level allows you to communicate the most useful information with the least amount of data going missing, hidden by overgeneralized aggregations or overlarge time spans, or hidden in the noise of overly-detailed time spans or too many split-bys.

Add context to the reports that you produce, providing details about the data analysis process and the dataset used, acknowledging what’s missing and what’s represented. Communicate missing data with detailed and focused visualizations, keeping visualizations consistent for regularly-communicated reports.

I hope that no matter your role in the data analysis process, this blog post series helps you reduce missing data and make smarter, more accurate, and less biased data-driven decisions.

Collect the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:36:39 +0000

This is the seventh post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

When you’re gathering the data you need and creating datasets that don’t exist yet, you’re in the midst of the data collection stage. Data can easily go missing when you’re collecting it!

In this post, I’ll cover the following:

How data goes missing at the data collection stage
What to do about missing data at the collection stage

How does data go missing?

There are many reasons why data might be missing from your analysis process. Data goes missing at the collection stage because the data doesn’t exist, or the data exists but you can’t use it for whatever reason, or the data exists but the events in the dataset are missing information.

The dataset doesn’t exist

Frequently data goes missing because the data itself does not exist, and you need to create it. It’s very difficult and impractical to create a comprehensive dataset, so data can easily go missing at this stage. It’s important to do what you can to make sure data goes consistently missing when you collect it, if possible, by collecting representative data.

In some cases, though, you do need comprehensive data. For example, if you need to create a dataset of all the servers in your organization for compliance reasons, you might discover that there is no one dataset of servers, and that efforts to compile one are a challenge. You can start with just the servers that your team administers, but that’s an incomplete list.

Some servers are grant-owned and fully administered by a separate team entirely. Perhaps some servers are lurking under the desks of some colleagues, connected to the network but not centrally managed. You can try to use network scans to come up with a list, but then you gather only those servers connected to the network at that particular time. Airgapped servers or servers that aren’t turned on 24/7 won’t be captured by such an audit. It’s important to continually consider if you really need comprehensive data, or just data that comprehensively addresses your use case.

The data exists, but…

There’s also a chance that the data exists, but isn’t machine-readable. If the data is provided only in PDFs, as many FOIA requests are returned in, then it becomes more difficult to include the data in data analysis. There’s also a chance that the data is available only as paper documents, as is the case with gun registration records. As Jeanne Marie Laskas reports for GQ in Inside The Federal Bureau Of Way Too Many Guns, having records only on paper prevents large-scale data analysis on the information, thus causing it to effectively go missing from the entire process of data analysis.

It’s possible that the data exists, but isn’t on the network—perhaps because it is housed on an airgapped device, or perhaps stored on servers subject to different compliance regulations than the infrastructure of your data analysis software. In this case, the data exists but it is missing from your analysis process because it isn’t available to you due to technical limitations.

Another common case is that the data exists, but you can’t have it. If you’ve made an enemy in another department, they might not share the data with you because they don’t want to. It’s more likely that access to the data is controlled by legal or compliance concerns, so you aren’t able to access the data for your desired purposes, or perhaps you can’t analyze it on the tool that you’re using for data analysis due to compliance reasons.

For example, most doctors offices and hospitals in the United States use electronic health records systems to store the medical records of thousands of Americans. However, scientific researchers are not permitted to access detailed electronic health records of patients, though they exist in large databases and the data is machine-readable, because the health insurance portability and accountability act (HIPAA) privacy rule regulates how protected health information (PHI) can be accessed and used.

Perhaps the data exists, but is only available to people who pay for access. This is the case for many music metadata datasets like those from Nielsen, much to my chagrin. The effort it takes to create quality datasets is often commoditized. This also happens with scientific research, which is often only available to those with access to scientific journals that publish the results of the research. The datasets that produce the research are also often closely-guarded, as one dataset is time-consuming to create and can lead to multiple publications.

There’s also a chance the data exists, but it isn’t made available outside of the company. A common circumstance for this is public API endpoints for cloud services. Spotify collects far more data than they make available via the API, so too do companies like Zoom or Google. You might hope to collect various types of data from these companies, but if the API endpoints don’t make the data available, you don’t have many options.

And of course, in some cases the data exists, but it’s inconsistent. Maybe you’re trying to collect equivalent data from servers or endpoints with different operating systems, but you can’t get the same details due to logging limitations. A common example is trying to collect the same level of detail from computers with MacOS and computers with Windows installed. You can also see inconsistencies if different log levels are set on different servers for the same software. This inconsistent data causes data to go missing within events and makes it more difficult to compare like with like.

14-page forms lead to incomplete data collection in a pandemic

Data can easily go missing if it’s just too difficult to collect. An example from Illinois, reported by WBEZ reporter Kristen Schorsch in Illinois’ Incomplete COVID-19 Data May Hinder Response, is that “the Illinois Department of Public Health issued a 14-page form that it has asked hospitals to fill out when they identify a patient with COVID-19. But faced with a cumbersome process in the midst of a pandemic, many hospitals aren’t completely filling out the forms.”

It’s likely that as a result of the length of the form, data isn’t consistently collected for all patients from all hospitals—which can certainly bias any decisions that the Illinois Department of Public Health might make, given that they have incomplete data.

In fact, as Schorsch reports, without that data, public health workers “told WBEZ that makes it harder for them to understand where to fight for more resources, like N95 masks that provide the highest level of protection against COVID-19, and help each other plan for how to make their clinics safer as they welcome back patients to the office.”

In this case, where data is going missing because it’s too difficult to collect, you can refocus your data collection on the most crucial data points for what you need to know, rather than the most complete data points.

What can you do about missing data?

Most crucially, identify the missing data. If you know that you need a certain type of data to answer the questions that you want to answer in your data analysis, you must know that it is missing from your analysis process.

After you identify the missing data, you can determine whether or not it matters. If the data that you do have is representative of the population that you’re making decisions about, and you don’t need comprehensive data to make those decisions, a representative sample of the data is likely sufficient.

Communicate your use cases

Another important thing you can do is to communicate your use cases to the people collecting the data. For example,

If software developers have a better understanding of how telemetry or log data is being used for analysis, they might write more detailed or more useful logging messages and add new fields to the telemetry data collection.
If you share a business case with cloud service providers to provide additional data types or fields via their API endpoints, you might get better data to help you perform less biased and more useful data analyses. In return, those cloud providers are likely to retain you as a customer.

Communicating the use case for data collection is most helpful when communicating that information leads to additional data gathering. It’s riskier when it might cause potential data sources to be excluded.

For example, if you’re using a survey to collect information about a population’s preferences—let’s say, the design of a sneaker—and you disclose that information upfront, you might only get people with strong opinions about sneaker design responding to your survey. That can be great if you want to survey only that population, but if you want a more mainstream opinion, you might miss those responses because the use case you disclosed wasn’t interesting to them. In that context, you need to evaluate the missing data for its relevance to your analysis.

Build trust when collecting sensitive data

Data collection is a trust exercise. If the population that you’re collecting data about does not understand why you’re collecting the data, or trust that you will protect it, use it as you say you will, or if they believe that you will use the data against them, you might end up with missing data.

Nowhere is this more apparent than with the U.S. Census. Performed every 10 years, the data from the census is used to determine political representation, distribute federal resources, and much more. Because of how the data from the census survey is used, a representative sample isn’t enough—it must be as complete a survey as possible.

The Census Bureau understands that mistrust is a common reason why people might not respond to the census survey. Because of that, the U.S. Census Bureau hires pollsters that are part of groups that might be less inclined to respond to the census, and also provide clear and easy-to-find details on their website (See How We Protect Your Information on census.gov) about the measures in place to protect the data collected in the census survey. Those details are even clear in the marketing campaigns urging you to respond to the census! The census survey also faces other challenges when ensuring the comprehensive survey is as complete as possible.

This year, the U.S. Census also faced time limits for completing the collecting and counting of surveys, in addition to delays already imposed by the COVID-19 pandemic. The New York Times has additional details about those challenges: The Census, the Supreme Court and Why the Count Is Stopping Early.

Address instances of mistrust with data stewardship

As Jill Lepore discusses in episode 4, Unheard, of her podcast The Last Archive, mistrust can also affect the accuracy of the data being collected, such as in the case of former enslaved people being interviewed by descendants of their former owners, or their current white neighbors, for records collected by the Works Progress Administration. Surely, data is missing from those accounts of slavery due to mistrust of the people doing the data collection, or at the least, because those collecting the stories perhaps do not deserve to hear the true lived experiences of the former enslaved people.

If you and your team are not good data stewards, if you don’t do a good job of protecting data that you’ve collected or managing who has access to that data, people are less likely to trust you with more data—and thus it’s likely that datasets you collect will be missing data. Because of that, it’s important to practice good data stewardship. Use datasheets for datasets, or a data biography to record when data was collected, for what purpose, by whom or what means, and more. You can then review those to understand whether data is missing, or even to remember what data might be intentionally missing.

In some cases, data can be intentionally masked, excluded, or left to collect at a later date. If you keep track of these details about the dataset during the data collection process, it’s easier to be informed about the data that you’re using to answer questions and thus use it safely, equitably, and knowledgeably.

Collect what’s missing, maybe

If possible and if necessary, collect the data that is missing. You can create a new dataset if one does not already exist, such as those that journalists and organizations such as Campaign Zero have been compiling about police brutality in the United States. Some data collection that you perform might supplement existing datasets, such as adding additional introspection details to a log file to help you answer a new question for an existing data source.

If there are cases where you do need to collect additional data, you might not be able to do so at the moment. In those cases, you can build a roadmap or a business case to collect the data that is missing, making it clear how it can help reduce uncertainty for your decision. That last point is key, because collecting more data isn’t always the best solution for missing data.

Sometimes, it isn’t possible to collect more data. For instance, if you’re trying to gather historical data, but everyone from that period has died and very few or no primary sources remain. Or cases where the data has been destroyed, such as in a fire or intentionally, as the Stasi did after the fall of the Berlin Wall.

Consider whether you need complete data

Also consider whether or not more data will actually help address the problem that you’re attempting to solve. You can be missing data, and yet still not need to collect more data in order to make your decision. As Douglas Hubbard points out in his book, How to Measure Anything, data analysis is about reducing uncertainty about what the most likely answer to a question is. If collecting more data doesn’t reduce your uncertainty, then it isn’t necessary.

Nani Jansen Reventlow of the Digital Freedom Fund makes this point clear in her Op-Ed on Al Jazeera, Data collection is not the solution for Europe’s racism problem. In that case, collecting more data, even though it could be argued that the data is missing, doesn’t actually reduce uncertainty about what the likely solution for racism is. Being able to quantify the effect or harms of racism on a region does not solve the problem—the drive to solve the problem is the only thing that can solve that problem.

Avoid cases where you continue to collect data, especially at the expense of an already-marginalized population, in an attempt to prove what is already made clear by the existing information available to you.

You might think that data collection is the first stage of a data analysis process, but in fact, it’s the second. The next and last post in this series covers defining the question that guides your data analysis, and how to take action to reduce bias in your data-driven decisions: Define the question: How missing data biases data-driven decisions.

Manage the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:35:03 +0000

This is the sixth post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

In this post, I’ll cover the following:

What is data management?
How does data go missing, featuring examples of disappearing data
What you can do about missing data

How you manage data in order to prepare it for analysis can cause data to go missing and decisions based on the resulting analysis to be biased. With so many ways for data to go missing, there’s just as many chances to address the potential bias that results from missing data at this stage.

What is data management?

Data management, for the purposes of this post, covers all the steps you take to prepare data after it’s been collected. That includes all the steps you take to answer the following questions:

How do you extract the data from the data source?
What transformations happen to the data to make it easier to analyze?
How is it loaded into the analysis tool?
Is the data normalized against a common information model?
How is the data structured (or not) for analysis?
What retention periods are in place for different types of data?
Who has access to the data?
How do people access the data?
For what use cases are people permitted to access the data?
How is information stored and shared about the data sources?
What information is stored or shared about the data sources?
What upstream and downstream dependencies feed into the data pipeline?

How you answer these questions (if you even consider them at all) can cause data to go missing when you’re managing data.

How does data go missing?

Data can go missing at this stage in many ways. With so many moving parts from various tooling and transformation steps being taken to prepare data for analysis and make it easier to work with, a lot can go wrong. For example, if you neglect to monitor your dependencies, a configuration change in one system can cause data to go missing from your analysis process.

Disappearing data: missing docs site metrics

It was just an average Wednesday when my coworker messaged me asking for help with her documentation website metrics search—she thought she had a working search, but it wasn’t showing the results she expected. It was showing her that no one was reading any of her documentation, which I knew couldn’t be true.

As I dug deeper, I realized the problem wasn’t the search syntax, but the indexed data itself. We were missing data!

I reported it to our internal teams, and after some investigation they realized that a configuration change on the docs site had resulted in data being routed to a different index. A configuration change that they thought wouldn’t affect anything ended up causing data to go missing for nearly a week because we weren’t monitoring dependencies crucial to our data management system.

Thankfully, the data was only misrouted and not dropped entirely, but it was a good lesson in how easily data can go missing at this management stage. If you identify the sources you expect to be reporting data, then you can monitor for changes in the data flow. You can also document those sources as dependencies, and ensure that configuration changes include additional testing to ensure the continued fidelity of your data collection and management process.

Disappearing data: data retention settings slip-up

Another way data can go missing is if you neglect to manage or be aware of default tool constraints that might affect your data.

In this example, I was uploading my music data to the Splunk platform for the first time. I was so excited to analyze the 10 years of historical data. I uploaded the file, set up the field extractions, and got to searching my data. I wrote an all time search to see how my music listening habits had shifted year over year in the past decade—but only 3 years of results were returned. What?!

In my haste to start analyzing my data, I’d completely ignored a warning message about a seemingly-irrelevant setting called max_days_ago. It turns out, this setting is set by default to drop any data older than 3 years. The Splunk platform recognized that I had data in my dataset older than 3 years, but I didn’t heed the warning and didn’t update the default setting to match my data. I ended up having to delete the data I’d uploaded, fix my configuration settings, and upload the data again—without any of it being dropped this time!

This experience taught me to pay attention to how I configure a tool to manage my data to make sure data doesn’t go missing. This happened to me while using the Splunk platform, but it can happen with whatever tool you’re using to manage, transform, and process your data.

As reported by Alex Hern in the Guardian,

“A million-row limit on Microsoft’s Excel spreadsheet software may have led to Public Health England misplacing nearly 16,000 Covid test results”. This happened because of a mismatch in formats and a misunderstanding of the data limitations imposed by the file formats used by labs to report case data, as well as of the software (Microsoft Excel) used to manage the case data. Hern continues, pointing out that “while CSV files can be any size, Microsoft Excel files can only be 1,048,576 rows long – or, in older versions which PHE may have still been using, a mere 65,536. When a CSV file longer than that is opened, the bottom rows get cut off and are no longer displayed. That means that, once the lab had performed more than a million tests, it was only a matter of time before its reports failed to be read by PHE.”

This limitation in Microsoft Excel isn’t the only way that tool limitations and settings can cause data to go missing at the data management stage.

Data transformation: Microsoft wants genes to be dates

If you’re not using Splunk for your data management and analysis, you might be using Microsoft Excel. It turns out that Microsoft Excel, despite (or perhaps due to) its popularity, can also cause data to go missing due to configuration settings. In the case of some genetics researchers, it turned out that Microsoft Excel was transforming their data incorrectly. The software was transforming certain gene names, such as MAR1 and DEC1, into dates of March 1 and December 1, causing data to go missing from the analysis.

Clearly, if you’re doing genetics research, this is a problem. Your data has been changed, and this missing data will bias any research based on this dataset, because certain genes are now dates!

To handle cases where a tool is improperly transforming data, you have 3 options:

Change the tool that you’re using,
Modify the configuration settings of the tool so that it doesn’t modify your data,
Or modify the data itself.

The genetics researchers ended up deciding to modify the data itself. The HUGO Gene Nomenclature Committee officially renamed 27 genes to accommodate this data transformation error in Microsoft Excel. Thanks to this decision, these researchers have one fewer configuration setting to worry about when helping to ensure vital data doesn’t go missing during the data analysis process.

What can you do about missing data?

These examples illustrate common ways that data can go missing at the management stage, but they’re not the only ways. What can you do when data goes missing?

Carefully set configurations

The configuration settings that you use to manage data that you’ve collected can result in events and data points being dropped.

For example, if you incorrectly configure data source collection, you might lose events or parts of events. Even worse, data can go missing if you incorrectly record events due to incorrect line breaking, truncation, time zone, timestamp recognition, or retention settings. Data can go missing inconsistently if all of the nodes of your data management system don’t have identical configurations.

You might cause some data to go missing intentionally. You might choose to drop INFO level log messages and collect only the ERROR messages in an attempt to track just the signal from the noise of log messages, or you might choose to drop all events older than 3 months from all data sources to save money on storage. These choices, if inadequately communicated or documented, can lead to false assumptions or incorrect analyses being performed on the data.

If you don’t keep track of configuration changes and updates, a data source format could change before you update the configurations to manage the new format, causing data to get dropped, misrouted, or otherwise go missing from the process.

If your data analyst is communicating their use cases and questions to you, you can better understand data retention settings according to those use cases, and review the current policies across your datasets and see how they compare for complementary data types.

You can also identify complementary data sources that might help the analyst answer the questions they want to answer, and plan how and when to bring in those data sources to improve the data analysis.

You need to manage dataset transformations just as closely as you do the configurations that manage the data.

Communicate dataset transformations

The steps you take to transform data can also lead to missing data. If you don’t normalize fields, or if your field normalizations are inconsistently applied across the data or across the data analysts, data can appear to be missing even if it is there. If some data has a field name of http_referrer and the same fields in other data sources are consistently http_referer, the data with http_referrer might appear to be missing for some data analysts when they start the data analysis process.

Normalization can also help you identify where fields might be missing across similar datasets, such as cases where an ID is present in one type of data but not another, making it difficult to trace a request across multiple services.

If the data analyst doesn’t know or remember which field name exists in one dataset, and whether or not it’s the same field as another dataset, data can go missing at the analysis stage—as we saw with my examples of the “rating” field missing from some events and the info field not having a value that I expected in the data analysis post from this series, Analyze the data: How missing data biases data-driven decisions.

In the same vein, if you use vague field names to describe the data that you’ve collected, or dataset names that ambitiously describe the data that you want to be collecting—instead of what you’re actually collecting—data can go missing. Shortcuts like “future-proofing” dataset names can be misleading to data analysts that want to easily and quickly understand what data they’re working with.

The data doesn’t go missing immediately, but you’re effectively causing it to go missing when data analysis begins if data analysts can’t correctly decipher what data they’re working with.

Educate and incorporate data analysis into existing processes

Another way data can go missing is painfully human. If the people that you expect to analyze the data and use it in their decision-making process don’t know how to use the tool that the data is stored in, well, that data goes missing from the process. Tristan Handy in the dbt blog post Analytics engineering for everyone discusses this problem in depth.

It’s important to not just train people on the tool that the data is stored in, but also make sure that the tool and the data in it are considered as part of the decision-making process. Evangelize what data is available in the tool, and make it easy to interact with the tool and the data. This is a case where a lack of confidence and knowledge can cause data to go missing.

Data gaps aren’t always caused by a lack of data—they can also be caused by knowledge gaps and tooling gaps if people aren’t confident or trained to use the systems with the data in them.

Monitor data strategically

Everyone wants to avoid missing data, but you can’t monitor what you can’t define. So in order to monitor data to prevent it from going missing, you must define what data you expect to see, both from which sources or at which ingestion volumes.

If you don’t have a way of defining those expectations, then you can’t alert on what’s missing. Start by identifying what you expect, and then quantify what’s missing based on those expectations. For guidance on how to do this in Splunk, see Duane Waddle’s blog post Proving a Negative, as well as the apps TrackMe or Meta Woot!.

Plan changes to the data management system carefully

It’s also crucial to review changes to the configurations that you use to manage data sources, especially changes to data structures or normalization in data sources. Make sure that you consistently deploy these changes as well, to reduce the chance that some sources collect different data in different ways from other sources for the same data.

Be careful to note downstream and upstream dependencies for your data management system, such as other tools, permissions settings, or network configurations, before making changes, such as an upgrade or a software change.

The simplest way for data to go missing from a data analysis process is when it’s being collected. The next post in the series discusses how data can go missing at the collection stage: Collect the data: How missing data biases data-driven decisions.

Analyze the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:34:53 +0000

This is the fifth post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

When you do data analysis, you’re searching and analyzing your data so that you can answer a specific question with the data so that you can make a decision at the end of the process. Unfortunately, there are many ways that data can go missing while you analyze it.

In this post, I’ll cover the following

Why disaggregation matters when performing data analysis
How data goes missing when analyzing data
What to do about missing data when analyzing data
How simple mistakes can cause data to go missing

Declining bird populations in North America

Any aggregation that you do in your data analysis can cause data to go missing. I recently finished listening to the podcast series The Last Archive, by Jill Lepore, and episode 9 talked about declining bird populations in North America. I realized that it’s an excellent example of why disaggregating data is vital to making sure that data doesn’t go missing during the analysis process.

As reported in Science Magazine, scientists have concluded that since 1970, the continent of North America has lost 3 billion birds—nearly 30% of the surveyed total. But what does that mean for specific bird populations? Are some more affected than others?

It’s easy for me to look around San Francisco and think, well clearly that many birds can’t be missing—I’m still surrounded by pigeons and crows, and ducks and geese are everywhere when I go to the park!

If you don’t disaggregate your results, there’s no way to determine which bird breeds are affected and where. You might assume that 30% of all bird breeds were equally affected across habitats and types. Thankfully, these scientists did disaggregate their results, so we can identify variations that otherwise would be missing from the analysis.

In the case of this study, we can see that some bird populations—Old world sparrows, Larks, and Starlings—are more affected than other types of birds, while others—Raptors, Ducks and geese, and Vireos—have flourished in the past 50 years.

Because the data is disaggregated, you can uncover data that would otherwise be missing from the analysis—how the different types of birds have actually been differently affected, due to habitat loss in grasslands, or cases where restoration and rehabilitation efforts have been effective, such as the resurgence in the population of raptors.

Without an understanding of which specific bird populations are affected, and where they live, you can’t take as effective action to help bird populations recover, because you’re missing too much data due to an overgeneralized aggregate. Any decisions you took based on the overgeneralized aggregate would be biased and ultimately incorrect.

In the case of this study, we know that targeted bird population restoration is perhaps most needed in grasslands habitats, like the Midwest where I grew up.

Unfortunately, the study only covers 76% of all bird breeds, so my city-dwelling self, will just continue to wonder how the bird population has changed since 1970 for pigeons, doves, crows, and others.

How does data go missing?

An easy way for data to go missing is for incomplete data to be returned when you’re analyzing the data. Many of these examples are Splunk-specific, but are limitations shared by most data analysis tools.

Truncated results

In Splunk, search results could be truncated for a variety of reasons. Truncated results have visible error messages if you’re running ad hoc searches, but you might not see error messages if the searches are producing scheduled reports.

If the indexers where the data is stored are unreachable, or your search times out before completing, the results could be incomplete. If you’re using a subsearch, you might hit the default event limit of 10K events, or the timeout limit of 60 seconds and have incomplete subsearch results. If you’re using the join search command, you might hit the 50K row limit that Nick Mealy discusses in his brilliant .conf talk, Master Joining Datasets Without Using Join.

If you’re searching an external service from Splunk, for example by using ldapsearch or a custom command to search an external database, you might not get complete results if that service is having problems or if you don’t have access to some data that you’re searching for.

It’s surprisingly easy for data to go missing when you’re correlating datasets.

Missing and “missing” fields across datasets

If you’re trying to compare datasets and some of the datasets are missing fields, you might accidentally miss data. Without the same field across multiple types of data, it can be difficult to perform valuable or accurate data correlations.

In this containerized cloud-native world, tracing outages across systems can be complex if you don’t have matching identifiers in every dataset that you’re using. As another example, it can be difficult to identify suspicious user session activity without the ability to correlate session identifiers with active users logged into a specific host.

Sometimes the fields aren’t actually missing, they’re just named differently. Because the data isn’t normalized, or the fields don’t match your naming expectations, they’re effectively missing from your analysis because you can’t find them.

Missing fields in a dataset that you want to include in your analysis

Sometimes data is missing from specific events within a dataset. For example, I wanted to determine the average rating that I gave songs in my iTunes library. However, iTunes XML files store tracks with no rating (or a zero star rating for my purposes) without a rating field at all in the events.

Calculating an average with that data missing gives me an average 3 star rating for all the tracks in my iTunes library.

But if I add the zero-star rated tracks back in, representing those as “tracks I have decided aren’t worthy of a rating” rather than “tracks that have not yet been rated”, the average changes to 2.5 stars per track.

If I’d used the results that were missing data, I’d have a biased interpretation of my average song rating in my iTunes library.

Mismatched numeric units and timezones

You might also have data that goes missing if you’re improperly analyzing data because you don’t know or misinterpret the units that the data is stored in.

Keep track of whether or not a field contains a time in minutes or seconds and if your network traffic logs are in bytes or megabytes. Data can vanish from your analysis if you improperly compare dissimilar units!

If you convert a time to a more human-readable format and make incorrect assumptions about the time format, such as the time zone that it’s in, you can cause data to go missing from the proper time period.

Even without transforming your data, if you’re comparing different datasets that store time data with different time zones, data can go missing. You might think that you’re comparing a four hour window from last Monday while debugging an issue across several datasets

It’s also important to note that how you choose to aggregate your data can hide data that is missing from your dataset, or hide data in a way that causes it to go missing.

Consider the granularity of your aggregations

When you perform your data analysis, consider what time ranges you’re using to bin or aggregate your data, the time ranges that you use to search your data across, for what data points, which fields you use to disaggregate your data, and how that might affect your results.

For example, it’s important to keep track of how you aggregate events across time periods in your analysis. If the timespans that you use when aggregating your data don’t align with your use case and the question you’re trying to answer, data can go missing.

In this example, I was trying to convert an existing search that showed me how much music I was listening to by type per year, into a search that would show me how much music I was listening to by weekday by year. Let’s take a look:

This was my initial attempt at modifying the data, and there’s a lot missing. There’s no results at all for Tuesdays in 2019, and the counts for Sundays in 2017, Mondays in 2018, and Thursdays in 2020 were laughably low. What did I do wrong?

It turned out that the time spans I was using in the original search to aggregate my data were too broad for the results I was trying to get. I was doing a timechart with a span of 1 month to start, and then trying to get data down to the a granularity of weekday. That wasn’t going to work!

I’d caused data to go missing because I didn’t align the question I was trying to answer with the time span aggregation in my analysis.

Thankfully, it was a quick fix. I updated the initial time grouping to match my desired granularity of one day, and I was no longer missing data from my results!

This is a case where an overly broad timespan aggregation, combined with a lack of consideration for my use case, caused data to go missing.

You make an error in your analysis process

How many of you have written a Splunk search with a conditional eval function and totally messed it up?

In this case, I wrote a search to calculate the duration of music listening activities—specifically, calculating an estimated amount of time spent at a music festival, DJ set, or concert—in order to compare how much time I spent listening to music before I was sheltering in place with after.

I used a conditional “info == concert” to apply an estimated concert length of 3 hours, but no field-value pair of info=concert existed in my data. In fact, concerts had no info field at all. It wasn’t until I’d finished my search, combining 2 other datasets, that I realized a concert I’d attended in March was missing from the results.

In order to prevent this data from going missing, I had to break my search down into smaller pieces, validating the accuracy of the results at each step against my expectations and knowledge of what the data should reflect. Eventually I realized that I’d caused data to go missing in my analysis process by making the assumption that an info=concert field existed in my data.

Ironically, this graph is still missing data because Last.fm failed to scrobble tracks from Spotify for several time periods in July and August.

What can you do about missing data?

If you have missing data in your analysis process, what can you do?

Optimize your searches and environment

If you’re processing large volumes of events, you can consider expanding limits for data, such as those for subsearches or timeouts for results. Default limits are the most common settings, and aren’t always something to adjust, but occasionally it makes sense to adjust the limits to suit your use cases if your architecture can handle it.

More practically, make sure that you’re running efficient and precise analyses to make the most use of your data analysis environment. If you’re using Splunk, I again recommend Nick Mealy’s excellent .conf talk, Master Joining Datasets Without Using Join for guidance on optimizing your searches, as well as Martin Müller’s talk on the Splunk Job Inspector, featuring Clara Merriman for the .conf20 edition.

Normalize your data

You can normalize your data to make it easier to use, especially if you’re correlating common fields across many datasets. Use field names that are as accurate and descriptive as possible, but also names that are consistent across datasets.

It’s a best practice to follow a common information model, but also ensure that you work closely with data administrators and data analysts (if you aren’t performing both roles) to make sure that analyst use cases and expectations align with administrator practices and implementations.

If you’re using Splunk, start with the Splunk Common Information Model (CIM) or write your own data model to help normalize fields. Keep in mind too, that you don’t have to use accelerated data models to use a data model when naming fields.

Enrich your datasets

You can also explore different ways to enrich your datasets earlier in the analysis process to help make sure data doesn’t go missing across datasets. If you perform data enrichment on the data as it streams in, using a tool like Cribl LogStream or Splunk Data Stream Processor, you can add this missing data back to the events and make it easier to accurately and usefully correlate data across datasets.

Consistently validate your results

Do what you can to consistently and constantly ensure the validity of your search results. Assess the validity against your expectations, but ultimately against itself and against context. If you don’t validate your results, you might lose data that is hidden inside of an aggregate or misinterpreted due to missing context.

Check different time ranges. What seems to be a gap or a pattern in data could just be seasonality or noise. Consider separating weekends from weekdays, or business hours from other hours, depending on the type of data that you’re aggregating.
Use different types of averages, and consider whether the span of values in your results are well-represented by an average. An average of a wide range of values can inaccurately reflect the range of the values, effectively hiding the minimum and the maximum values.
Disaggregate your data to identify where you might be missing more data, and thus have bias in your results. Heather Krause and Shena Ashley discuss disaggregation in an interview for If Data Could Talk by Tableau Software on The Ethics of Visualizing Data on Race, making the point that disaggregated data does not imply causation, it merely describes the data.
Compare like with like. Validate time zones, units, and the data in similarly-named fields using a data dictionary or by working with the stewards of the datasets to make sure that you’re using the data correctly.

To help ensure you’re properly accounting for missing data that you might cause while analyzing data, work with your data administrator and consult with a statistics expert if you’re not sure you’re properly analyzing or aggregating data. To help learn more, I recommend Ben Jones’ book Avoiding Data Pitfalls: How to Steer Clear of Common Blunders When Working with Data and Presenting Analysis and Visualizations.

The next post in this series covers the different ways that data can go missing at the data management stage—what happens to prepare data for analysis? Manage the data: How missing data biases data-driven decisions.

Visualize the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:33:26 +0000

This is the fourth post in a series about how missing data can bias data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

Visualizing data is crucial to communicate the results of a data analysis process. Whether you use a chart, a table, a list of raw data, or a three-dimensional graph that you can interact with in virtual reality—your visualization choice can cause data to go missing. Any time you visualize the results of data analysis, you make intentional decisions about what to visualize and what not to visualize. How can you make sure that data that goes missing at this stage doesn’t bias data-driven decisions?

In this post, I’ll cover the following:

How the usage of your data visualization can cause data to go missing
How data goes missing in data visualizations
Why accessibility matters for data visualizations
How a lack of labels and scale can mislead and misinform
What to do about missing data in data visualizations

How people use the Georgia Department of Public Health COVID-19 daily report

When creating a data visualization, it’s important to consider how it will be used. For example, the state of Georgia provides a Department of Public Health Daily COVID-19 reporting page to help communicate the relative case rate for each county in the state.

In the midst of this global pandemic, I’m taking extra precautions before deciding to go hiking or climbing outside. Part of that risk calculation involves checking the relative case rate in my region — are cases going up, down, or staying the same?

If you wanted to check that case rate in Georgia in July, you might struggle to make an unbiased decision about your safety because of the format of a data visualization in that report.

As Andisheh Nouraee illustrates in a now-deleted Twitter thread, the Georgia Department of Public Health on the COVID-19 Daily Status Report provided a heat map in July that visualized the number of cases across counties in Georgia in such a way that it effectively hid a 49% increase in cases across 15 days.

Image from July 2nd, shared by Andisheh Nouraee, my screenshot of that image

Image from July 17th, shared by Andisheh Nouraee, my screenshot of that image

You might think that these visualizations aren’t missing data at all—the values of the gradient bins are clearly labeled, and the map clearly shows how many cases exist for every 100K residents.

However, the missing data isn’t in the visualization itself, but in how it’s used. This heat map is provided to help people understand the relative case rate. If I were checking this graph every week or so, I would probably think that the case rate has stayed the same over that time period.

Instead, because the visualization uses auto-adjusting gradient bins, the red counties in the visualization from July 2nd cover a range from 2961 to 4661, while the same color counties on July 17th now have case rates of 3769–5165 cases per 100K residents. The relative size of the bins is different enough to where the bins can’t be compared with each other over time.

As reported by Keren Landman for the Atlanta Magazine, the Department of Public Health didn’t have direct control over the data on the dashboard anyway, making it harder to make updates or communicate the data more intentionally.

Thankfully, the site now uses a visualization with a consistent gradient scale, rather than auto-adjusting bins.

In this example, the combination of the visualization choice and the use of that visualization by the visitors of this website caused data to go missing and possibly resulting in biased decisions about whether it’s safe to go for a hike in the community.

How does data go missing?

This example from the Georgia Department of Health describes one way that data can go missing, but there are many more.

Data can go missing from your visualization in a number of ways:

If the data exists, but is not represented in the visualization, data is missing
If data points and fluctuations are smoothed over, or connected across gaps, data is missing.
If outliers and other values are excluded from the visualization, data is missing.
If people can’t see or interact with the visualization, data is missing.
If a limited number of results are being visualized, but the label and title of the visualization don’t make that clear, data is missing.

Accessible data visualizations prevent data from going missing

Accessible visualizations are crucial for avoiding missing data because data can go missing if people can’t see or interact with it.

Lisa Charlotte Rost wrote an excellent series for Data Wrapper’s blog about colorblindness and data visualizations that I highly recommend for considering color vision accessibility for data visualization: How your colorblind and colorweak readers see your colors, What to consider when visualizing data for colorblind readers, and What’s it like to be colorblind.

You can also go further to consider how to make it easier for folks with low or no vision to interact with your data visualizations. Data visualization artist Mona Chalabi has been experimenting with ways to make her data visualization projects more accessible, including making a tactile version of a data visualization piece, and an interactive piece that uses touch and sound to communicate information, created in collaboration with sound artist Emmy the Great.

At a more basic level, consider how your visualizations look at high zoom levels and how they sound when read aloud by a screen reader. If a visualization is unintelligible at high zoom levels or if portions aren’t read aloud by a screen reader, those are cases where data has gone missing from your visualization. Any decisions that someone with low or no vision wants to make based on a data visualization is biased to include only the data visualizations that they can interact with successfully.

Beyond vision considerations, you want to consider cognitive processing accessibility to prevent missing data. If you overload a visualization with lots of overlays, rely on legends to communicate meaning in your data, or have a lot of text in your visualization, folks with ADHD or dyslexia might struggle to process your visualization.

Any data that people can’t understand in your visualization is missing data. For more, I recommend the blog post by Sarah L. Fossheim, An intro to designing accessible data visualizations.

Map with caution and label prodigiously: Beirut explosion map

Data can go missing if you fail to visualize it clearly or correctly. When I found out about the explosion in Beirut, after I made sure that my friends and their family were safe, I wanted to better understand what had happened.

Image shared by Joanna Merson, my screenshot of the image

I haven’t had the privilege to visit Beirut before, so the maps of the explosion radius weren’t as easy for me to personally relate to. Thankfully, people started sharing maps about what the same explosion might look like if it occurred in New York City or London.

Image shared by Joanna Merson, my screenshot of the image

This map attempts to show the scale of the same explosion in New York City, but it’s missing a lot of data. I’m not an expert in map visualizations, but thankfully cartographer Joanna Merson tweeted a correction to this map and unpacked just how much data is missing from this visualization.

There’s no labels on this map, so you don’t know the scale of the circles, or what distance each blast radius is supposed to represent. You don’t know what the epicenter of the blast is because it isn’t labeled, and perhaps most egregiously, the map projection used is incorrect.

Joanna Merson created an alternate visualization, with all the missing data added back in.

Image by Joanna Merson, my screenshot of the image.

Her visualization carefully labels the epicenter of the blast, as well as the radii of each of the circles that represent different effects from the blast. She’s also careful to share the map projection that she used—one that has the same distance for every point along that circle. It turns out that the projection used by Google Maps is not the right projection to show distance with an overlaid circle. Without the scale or an accurate projection in use, data goes missing (and gets added) as unaffected areas are misleadingly shown as affected by the blast.

How many of you are guilty of making a geospatial visualization, but don’t know anything about map projections and how they might affect your visualization?

Joanna Merson further points out in her thread on Twitter that maps like this with an overlaid radius to show distance can be inaccurate because they don’t take into account the effect of topography. Data goes missing because topography isn’t represented or considered by the visualization overlaid on the map.

It’s impractical to model everything perfectly in every map visualization. Depending on how you’re using the map, this missing data might not actually matter. If you communicate what your visualization is intended to represent when you share it, you can convey the missing data and also assert its irrelevance to your point. All maps, after all, must make decisions about what data to include based on the usage of the map. Your map-based data visualizations are no different!

It can be easy to cut corners and make a simple visualization to communicate the results of data analysis quickly. It can be tedious to add a scale, a legend, and labels to your visualization. But you must consider how your visualization might be used after you make it—and how it might be misused.

Will a visualization that you create end up in a blog post like this one, or a Twitter thread unpacking your mistakes?

What can you do about missing data?

To prevent or mitigate missing data in a data visualization, you have several options. Nathan Yau of Flowing Data has a very complete guide for Visualizing Incomplete and Missing Data that I highly recommend in addition to the points that I’m sharing here.

Visualize what’s missing

One important way to mitigate missing data in a data visualization is to devise a way to show the data that is there alongside the data that isn’t. Make the gaps apparent and visualize missing data, such as by avoiding connecting the dots between missing values in a line chart.

In cases where your data has gaps, you can add annotations or labels to acknowledge and explain any inconsistencies or perceived gaps in the data. In some cases, data can appear to be missing, but is actually a gap in the data due to seasonal fluctuations or other reasons. It’s important to thoroughly understand your data to identify the difference.

If you visualize the gaps in your data, you have the opportunity to discuss what can be causing the gaps. Gaps in data can reflect reality, or flaws in your analysis process. Either way, visualizing the gaps in your data is just as valuable as visualizing the data that you do have. Don’t hide or ignore missing data.

Carefully consider time spans

Be intentional about the span that you choose for time-based visualizations. You can unintentionally hide fluctuations in the data if you choose an overly-broad span for your visualization, causing data to go missing by flattening it.

If you choose an overly-short time span for your visualization, however, the meaning of the data and what you’re trying to communicate can go missing with all the noise of the individual data points. Consider what you’re trying to communicate with the data visualization, and choose a time span accordingly.

Write clearly

Another way to address missing data is to write good labels and titles for visualizations. It’s crucial to explain exactly what is present in a visualization—an important component of communicating results. If you’re intentional and precise about your labels and titles, you can prevent data from going missing.

If the data analysis contains the results for the top 10 cities by population density, but your title only says “Top Cities”, data has gone missing from your visualization!

You can test out the usefulness of your labels and titles by considering the following: If someone screenshots your visualization and puts it in a different presentation, or tweets it without the additional context that might be in the full report, how much data would be missing from the visualization? How completely does the visualization communicate the results of data analysis if it’s viewed out of context?

Validate your scale

Make sure any visualization that you create has a scale and that it’s included. It’s really easy for data to go missing if the scale of the data itself is missing.

Also validate that the scale on your visualization is accurate and relevant. If you’re visualizing percentages, make sure the scale goes from 0-100. If you’re visualizing logarithmic data, make sure your scale reflects that correctly.

Consider the use

Consider how your visualization will be used, and design your visualizations accordingly. What decisions are people trying to make based on your visualization? What questions are you trying to answer when you make it?

Automatically-adjusting gradient bins in a heat map can be an excellent design choice, but as we saw in Georgia, they don’t make sense to communicate relative change over time.

Choose the right chart for the data

It’s also important to choose the right chart to visualize your data. I’m not a visualization expert, so check out this data tutorial from Chartio, How to Choose the Right Data Visualization as well as these tutorials of different chart types on Flowing Data: Chart Types.

I do want to recommend that if you’re visualizing multiple aggregations in one visualization in the Splunk platform, consider the Trellis layout to create different charts to help compare across the aggregates.

Always try various types of visualizations for your data to determine which one shows the results of your analysis in the clearest way.

One of the best ways to make sure your data visualization isn’t missing data is to make sure that the data analysis is sound.

The next post in this series addresses how data can go missing while you analyze it: Analyze the data: How missing data biases data-driven decisions.

Communicate the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:32:00 +0000

This is the third post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

Communicating the results of a data analysis process is crucial to making a data-driven decision. You might review results communicated to you in many ways:

A slide deck presented to you
An automatically-generated report emailed to you regularly
A white paper produced by an expert analysis firm that you review
A dashboard full of curated visualizations
A marketing campaign

If you’re a data communicator, what you choose to communicate (and how you do so) can cause data to go missing and possibly bias data-driven decisions made as a result. In this post, I’ll cover the following:

A marketing campaign that misrepresents the results of a data analysis process
A renowned white paper produced by leaders in the security space
How data goes missing at the communication stage
What you can do about missing data at the communication stage

Spotify Wrapped: Your “year” in music

If you listen to music on Spotify, you might have joined in the hordes of people that viewed and shared their Spotify Wrapped playlists and images like these last year.

Spotify Wrapped is a marketing campaign that purports to communicate your year in music, based on the results of some behind-the-scenes data analysis that Spotify performs. While it is a marketing campaign, it’s still a way that data analysis results are being communicated to others, and thus relevant to this discussion of missing data in communications.

In this case you can see that my top artists, top songs, minutes listened, and top genre are shared with me, along with the number of artists that I discovered, and even the number of minutes that I spent listening to music over the last year. It’s impressive, and exciting to see my year in music summed up in such a way!

But after I dug deeper into the data, the gaps in the communication and data analysis became apparent. What’s presented as my year in music is actually more like my “10 months in music”, because the data only represents the period from January 1st to October 31st of 2019. Two full months of music listening behavior are completely missing from my “year” in music.

Unfortunately, these details about the dataset are missing from the Spotify Wrapped campaign report itself, so I had to search through additional resources to find out more information. According to a now-archived FAQ on Spotify for Artists (thank goodness for the Wayback Machine), the data represented in the campaign covers the dates from January 1st, 2019 – October 31st, 2019. I also read two key blog posts, Spotify Wrapped 2019 Reveals Your Streaming Trends, from 2010 to Now announcing the campaign and Unwrapping Wrapped 2019: Spotify VP of Engineering Tyson Singer Explains digging into the data analysis behind the campaign, to learn what I could about the size of the dataset, or how many data points might be necessary to draw the conclusions shared in the report.

This is a case where data is going missing from the communication because the data is presented as though it represents an entire time period, when in fact it only represents a subset of the relevant time period. Not only that, but it’s unclear what other data points actually represent. The number of minutes spent listening to music in those 10 months could be calculated by adding up the actual amount of time I spent listening to songs on the service, but could also be an approximate metric calculated from the number of streams of tracks in the service. It’s not possible to find out how this metric is being calculated. If it is an approximate metric based on the number of streams, that’s also a case of uncommunicated missing data (or a misleading metric), because according to the Spotify for Artists FAQ, Spotify counts a song as streamed if you’ve listened to at least 30 seconds of a track.

It’s likely that other data is missing in this incomplete communication, but another data communication is a great example of how to do it right.

Verizon DBIR: A star of data communication

The Verizon Data Breach Investigations Report (DBIR) is an annual report put out by Verizon about, well, data breach investigations. Before the report shares any results, it includes a readable summary about the dataset used, how the analysis is performed, as well as what is missing from the data. Only after the limitations of the dataset and the resulting analysis is communicated, are the actual results of the analysis shared.

And those results are well-presented, featuring confidence intervals, detailed titles and labels, as well as clear scales to visualize the data. I talk more about how to prevent data from going missing in visualizations in the next post of the series, Visualize the data: How missing data biases data-driven decisions.

How does data go missing?

These examples make it clear that data can easily go missing when you’re choosing how to communicate the results of data analysis.

You include only some visualizations in the report — the prettiest ones, or the ones with “enough” data
You include different visualizations and metrics than the ones in the last report, without an explanation about why they changed, or what changed. For example, what happened in Spain in late May, 2020 as reported by the Financial Times: Flawed data casts cloud over Spain’s lockdown strategy.
You choose not to share or discuss the confidence intervals for your findings.
You neglect to include details about the dataset, such as the size of the dataset or the representativeness of the dataset.
You discuss the available data as though it represents the entire problem, rather than a subset of the problem.

For example, the Spotify Wrapped campaign shares the data that it makes available as though it represents an entire year, but instead only reflects 10 months, and doesn’t include any details about the dataset beyond what you can assume—it’s based on Spotify’s data. This missing data doesn’t make the conclusions drawn in the campaign inaccurate, but it is additional context that can affect how you interpret the findings and make decisions based on the data, such as which artist’s album you might buy yourself to celebrate the new year.

What can you do about missing data?

To mitigate missing data in your communications, it’s vital to be precise and consistent. Communicate exactly what is and is not covered in your report. If it makes sense, you could even share the different levels of disaggregation used throughout the data analysis process that led to the results discussed in the final communication.

Consistency in the visualizations that you choose to include, as well as the data spans covered in your report, can make it easier to identify when something has changed since the last report. Changes can be due to missing data, but can also be due to something else that might not be identified if the visualizations in the report can’t be compared with each other. If you do change the format of the report, consider continuing to provide the old format alongside the new format. Alternately, highlight what is different from the last report, why you made changes, and clearly discuss whether comparisons can or cannot be made with older formats of the report to avoid unintentional errors.

If you know data is missing, you can discuss it in the report, share why or why not the missing data matters, and possibly choose to communicate a timeline or a business case for addressing the missing data. For example, there might be a valid business reason why data is missing, or why some visualizations have changed from the previous report. The 2020 edition of Spotify Wrapped will almost certainly include different types of data points due to the effects of the global pandemic on people’s listening habits. Adding context to why data is missing, or why the report has changed, can add confidence in the face of missing data—people now understand why data is missing or has changed.

Often when you’re communicating data, you’re including detailed visualizations of the results of data analysis processes.

The next post in this series covers how data can go missing from visualizations, and what to do about it: Visualize the data: How missing data biases data-driven decisions.

Decide with the data: How missing data biases data-driven decisions

Mon, 26 Oct 2020 16:31:37 +0000

This is the second post in a series about how missing data biases data-driven decisions. Start at the beginning: What’s missing? Reduce bias by addressing data gaps in your analysis process.

Any decision based solely on the results of data analysis is missing data—the non-quantitative kind. But data can also go missing from data-driven decisions as a result of the analysis process. Exhaustive data analysis and universal data collection might seem like the best way to prevent missing data, but it’s not realistic, feasible, or possible. So what can you do about the possible bias introduced by missing data?

In this post, I’ll cover the following:

What to do if you must make a decision with missing data
How data goes missing at the decision stage
What to do about missing data at the decision stage
How much missing data matters
How much to care about missing data before making your decision

Missing data in decisions from the Oregon Health Authority

In the midst of a global pandemic, we’ve all been struggling to evaluate how safe it is to resume pre-pandemic activities like going to the gym, going out to bars, sending our kids back to school, or eating at restaurants. In the United States, state governors are the ones tasked with making decisions about what to reopen and what to keep closed, and most are taking a data-driven approach.

In Oregon, the decisions they’re making about what to reopen and when are based on incomplete data. As reported by Erin Ross for Oregon Public Broadcasting, the Oregon Health Authority is not collecting or analyzing data about whether or not restaurants or bars are contributing to COVID-19 case rates.

The contact tracers interviewing people who’ve tested positive for SARS COV-2 are not asking, and even if the information is shared, the data isn’t being analyzed in a way that might allow officials to identify the effect of bars and restaurants on coronavirus case rates. Although this data is missing, officials are making decisions about whether or not bars and restaurants should remain open for indoor operations.

Oregon and the Oregon Health Authority aren’t alone in this limitation. We’re in the midst of a pandemic, and everyone is doing the best they can with the limited resources and information that they have. Complete data isn’t always possible, especially when real-time data matters. So what can Oregon (and you) do to make sure that missing data doesn’t negatively affect the decision being made?

If circumstances allow, it’s best to try to narrow the scope of your decision. Limit your decision to those groups and situations about which you have complete or representative data. If you can’t limit your decision, such as is the case with this pandemic, you can still make a decision with incomplete data.

Acknowledge that your decision is based on limited data, identify the gaps in your knowledge, and make plans to address those gaps as soon as possible. You can address the missing data by collecting more data, analyzing your existing data differently, or by reexamining the relevance of various data points to the decision that you’re making. This is one key example of how missing data can affect a decision-making process. How else can data go missing when making decisions?

How can data go missing?

Data-driven decisions are especially vulnerable to the effects of missing data. Because data-driven are based on the results of data analysis, the effects of missing data in the earlier data analysis stages are compounded.

The reports that you reviewed before making your decision included the prettiest graphs instead of the most useful visualizations to help you make your decision.
The visualizations in the report were different from the ones in the last report, making it difficult for you to compare the new results with the results in the previous report.
The data being collected doesn’t include the necessary details for your decision. This is what is happening in Oregon, where the collected data doesn’t include all the details that are relevant when making decisions about what businesses and organizations to reopen.
The data analysis that was performed doesn’t actually answer the question that you’re asking. If you need to know “how soon can we reopen indoor dining and bars despite the pandemic”, and the data analysis being performed can only tell you “what are the current infection rates, by county, based on the results of tests administered 5 days ago”, the decisions that you’re making might be based on incomplete data.

What can you do about missing data?

Identify if the missing data matters to your decision. If it does, you must acknowledge that the data is missing when you make your decision. If you don’t have data about a group, intentionally exclude that group from your conclusions or decision-making process. Constrain your decision according to what you know, and acknowledge the limitations of the analysis.

If you want to make broader decisions, you must address the missing data throughout the rest of the process! If you aren’t able to immediately collect missing data, you can attempt to supplement the data with a dedicated survey aimed at gathering that information before your decision. You can also investigate to find out if the data is already available in a different format or context—for example in Oregon, where the Health Authority might have some information about indoor restaurant and bar attendance in the content of contact tracing interviews and just aren’t analyzing it systematically. If the data is representative even if it isn’t comprehensive, you can still use it to supplement your decision.

To make sure you’re making an informed decision, ask questions about the data analysis process that led to the results you’re reviewing. Discuss whether or not data could be missing from the results of the analysis presented to you, and why. Ask yourself: does the missing data affect the decision that I’m making? Does it affect the results of the analysis presented to me? Evaluate how much missing data matters to your decision-making process.

You don’t always need more data

You will always be missing some data. It’s important to identify when the data that is missing is actually relevant to your analysis process, and when it won’t change the outcome. Acknowledge when additional data won’t change your conclusions.

You don’t need all possible data in existence to support a decision. As Douglas Hubbard points out in his book How to Measure Anything, the goal of a data analysis process is to reduce your uncertainty about the right approach to take.

If additional data, or more detailed analysis, won’t further reduce any uncertainty, then it’s likely unnecessary. The more clearly you constrain your decisions, and the questions you use to guide your data analysis, the more easily you can balance reducing data gaps and making a decision with the data and analysis results you have.

USCIS doesn’t allow missing data, even if it doesn’t affect the decision

Sometimes, missing data doesn’t affect the decision that you’re making. This is why you must understand the decision you’re making, and how important comprehensive data is to the decision. If that’s true, you want to make sure your policies acknowledge that reality.

In the case of the U.S. Citizenship and Immigration Services (USCIS), their policies don’t seem to recognize that some kinds of missing data for citizenship applications are irrelevant.

In an Opinions column by Washington Post reporter Catherine Rampell, The Trump administration’s no-blanks policy is the latest Kafkaesque plan designed to curb immigration, she describes the “no blanks” policy applied to immigration applications, and now, to third-party documents included with the applications.

“Last fall, U.S. Citizenship and Immigration Services introduced perhaps its most arbitrary, absurd modification yet to the immigration system: It began rejecting applications unless every single field was filled in, even those that obviously did not pertain to the applicant. “Middle name” field left blank because the applicant does not have a middle name? Sorry, your application gets rejected. No apartment number because you live in a house? You’re rejected, too. No address given for your parents because they’re dead? No siblings named because you’re an only child? No work history dates because you’re an 8-year-old kid? All real cases, all rejected.”

In this example, missing data is deemed a problem for making a decision about the citizenship application for a person—even when the data that is missing is supposed to be missing because it doesn’t exist. When asked for comment,

“a USCIS spokesperson emailed, “Complete applications are necessary for our adjudicators to preserve the integrity of our immigration system and ensure they are able to confirm identities, as well as an applicant’s immigration and criminal history, to determine the applicant’s eligibility.””

Missing data alone is not enough to affect your decision—only missing data that affects the results of your decision. A lack of data is not itself a problem—the problem is when that is relevant to your decision is missing. That’s how bias gets introduced to a data-driven decision.

In the next post in this series, I’ll explore some ways that data can go missing when the results of data analysis are communicated: Communicate the data: How missing data biases data-driven decisions.

What’s missing? Reduce bias by addressing data gaps in your analysis process

Mon, 26 Oct 2020 16:30:30 +0000

We live in an uncertain world. Facing an ongoing global pandemic, worsening climate change, persistent threats to human rights, and the more mundane uncertainties of our day-to-day lives, we try to use data to make sense of it all. Relying on data to guide decisions can feel safe.

But you might not be able to trust your data-driven decisions if data is missing from your data analysis process. If you can identify and address gaps in your data analysis process, you can reduce the bias introduced by missing data in your data-driven decisions, regaining your confidence and certainty while ensuring you limit possible harm.

This is post 1 of 8 in a series about how missing data can negatively affect a data analysis process.

In this post, I’ll cover:

What is missing data?
Why missing data matters
What’s missing from all data-driven decisions
The stages of a data analysis process

I hope this series inspires you and prepares you to take action to address bias introduced by missing data in your own data analysis processes. At the very least, I hope you gain a new perspective when evaluating your data-driven decisions, the success of data analysis processes, and how you frame a data analysis process from the start.

What is missing data?

Data can go missing in many ways. If you’re not collecting data, or don’t have access to some kinds of data, or if you can’t use existing data for a specific data analysis process—that data is missing from your analysis process.

Other data might not be accessible to you for other reasons. Throughout this series, I’ll use the term “missing data” to refer to both data that does not exist, data that you do not have access to, and data that is obscured by your analysis process—effectively missing, even if not literally gone.

Why missing data matters

Missing data matters because it can easily introduce bias into the results of a data analysis process. Biased data analysis is often framed in the context of machine learning models, training datasets, or inscrutable and biased algorithms leading to biased decisions.

But you can draw biased and inaccurate conclusions from any data analysis process, regardless of whether machine learning or artificial intelligence is involved. As Meg Miller makes clear in her essay Finding the Blank Spots in Data for Eye on Design, “Artists and designers are working to address a major problem for marginalized communities in the data economy: “If the data does not exist, you do not exist.””. And that’s just one part of why missing data matters.

You can identify the possible biases in your decisions if you can identify the gaps in your data and data analysis process. And if you can recognize those possible biases, you can do something to mitigate them. But first we need to acknowledge what’s missing from every data-driven decision.

What’s missing from all data-driven decisions?

It feels safe to make a data-driven decision. You’ve performed a data analysis process and have a list of results matched up with objectives that you want to achieve. It’s easy to equate data with neutral facts. But we can’t actually use data for every decision, and we can’t rely only on data for a decision-making process. Data can’t capture the entirety of an experience—it’s inherently incomplete.

Data only represents what can be quantified. Howard Zinn writes about the incompleteness of data in representing the horrors of slavery in A People’s History of the United States:

“Economists or cliometricians (statistical historians) have tried to assess slavery by estimating how much money was spent on slaves for food and medical care. But can this describe the reality of slavery as it as to a human being who lived in side it? Are the conditions of slavery as important as the existence of slavery?” “But can statistics record what it meant for families to be torn apart, when a master, for profit, sold a husband or a wife, a son or a daughter?” (pg 172, emphasis original).

Statistical historians and others can attempt to quantify the effects of slavery based on the records available to them, but parts of that story can never be quantified. The parts that can’t be quantified must be told, and must be considered when creating a historical record and of course, in deciding whose story gets told and how.

What data is available, and from whom, represents an implicit value and power structure in society as well. If data has been collected about something, and made available to others, then that information must be important—whether to an organization, a society, a government, or a world—and the keepers of the data had the privilege and the power to maintain it and make it available after it was collected.

This power structure, this value structure, and the limitations of data alone when making decisions are crucial to consider in this era of seemingly-objective data-driven decisions. Because data alone isn’t enough to capture the reality of a situation, it isn’t enough to drive the decisions you make in our uncertain world. And that’s only the beginning of how missing data can affect decisions.

Data can go missing at any stage of the data analysis process

It’s easy to consider missing data as solely a data collection problem—if the dataset existed, or new data was collected, no data would be missing and so we can make better data-driven decisions. In fact, avoiding missing data when you’re collecting it is just one way to reduce bias in your data-driven decisions—it’s far from the only way.

Data can go missing at any stage of the data analysis process and bias your resulting decisions. Each post in this series addresses a different stage of the process.

🗣 Make a decision based on the results of the data analysis. Decide with the data: How missing data biases data-driven decisions.
📋 Communicate the results of the data analysis. Communicate the data: How missing data biases data-driven decisions.
📊 Visualize the data to represent the answers to your questions. Visualize the data: How missing data biases data-driven decisions.
🔎 Analyze the data to answer your questions. Analyze the data: How missing data biases data-driven decisions.
🗂 Manage the data that you’ve collected to make it easier to analyze. Manage the data: How missing data biases data-driven decisions.
🗄 Collect the data you need to answer the questions you’ve defined. Collect the data: How missing data biases data-driven decisions.
🙋🏻‍♀️ Define the question that you want to ask the data. Define the question: How missing data biases data-driven decisions.

In each post, I’ll discuss real world examples of how data can go missing, and what you can do about it!

How do you make large scale harm visible on the individual level?

Tue, 29 Sep 2020 10:11:47 +0000

Teams that build security and privacy tools like Brave Browser, Tor Browser, Signal, Telegram, and others focus on usability and feature parity of these tools in an effort to more effectively acquire users from Google Chrome, iMessage, Google Hangouts, WhatsApp, and others.

Do people fail to adopt these more secure and private tools because they aren’t as usable as what they’re already using, or because it requires too much effort to switch?

I mean, of course it’s both. You need to make the effort to switch, and in order to switch you need viable alternatives to switch to. And that’s where the usability and feature parity of Brave Browser and Signal compared with Google Chrome and WhatsApp come in.

But if we’re living in a world where feature parity and usability are a foregone conclusion, and we are, then what? What needs to happen to drive a large-scale shift away from data-consuming and privacy-invading tools and toward those that don’t collect data and aggressively encrypt our messages?

To me, that’s where it becomes clear that the amorphous effects of widespread data collection—though well-chronicled in blog posts, books, and shows like The Social Dilemma— don’t often lead to real change unless a personal threat is felt.

Marginalized and surveilled communities adopt tools like Signal or FireChat in order to protect their privacy and security, because their privacy and security are actively under threat. For others, their privacy and security is still under threat, but indirectly. Lacking a single (or a series of) clear events that are tied to direct personal harm, people don’t often abandon a platform.

If I don’t see how the use of using Google Chrome, YouTube, Facebook, Instagram, Twitter, and other sites and tools cause direct harm to me, I have little incentive to make a change, despite the evidence of aggregate harm on society—amplified societal divisions, active disinformation campaigns, and more.

Essays that expose the “dark side” of social media and algorithms make an attempt to identify distinct personal harms caused by these systems. Essays like James Bridle’s essay on YouTube, Something is wrong on the internet (2017), or Adrian Chen’s essay about what social media content moderators experience, The Laborers Who Keep Dick Pics and Beheadings Out of Your Facebook Feed (2014) or Casey Newton’s about the same, The secret lives of Facebook moderators in America (2019), gain widespread attention for the problems they expose, but don’t necessarily lead to people abandoning the platforms, nor lead the platforms themselves to take action.

These theorists and journalists are making a serious attempt to make large-scale harm caused by these platforms visible on an individual level, but nothing is changing. Is it the fault of the individual, or the platform?

Spoilers, it’s always “both”. And here we can draw an analogy to climate change too. As with climate change, the effects resulting from these platforms and companies are so amorphous, it’s possible to point to alternate explanations—for a time. Dramatically worsening wildfires in the Western United States are a result of poor fire policy, worsening tropical storms are a result of weaker wind patterns (or stronger ones? I don’t study wind).

One could argue that perhaps climate change is the result of mechanization and industrialization in general, and it would be happening without the companies currently contributing to it. Perhaps the dark side of the internet is just the dark side of reality, and nothing worse than would exist without these platforms and companies contributing.

The truth is, it’s both. We live in a “yes, and” world. Climate change is causing, contributing to, and intensifying the effects of wildfires and the strength and frequency of tropical storms and hurricanes. Platform algorithms are causing, contributing to, and intensifying the effects of misinformation campaigns and violence on social media and the internet.

And much like companies that contributed to climate change knew what was happening, as reported in The Guardian: Shell and Exxon’s secret 1980s climate change warnings, Facebook Google and others know that their algorithms are actively contributing to societal harm—but the companies aren’t doing enough about it.

So what’s next?

Do we continue to attempt to make the individual feel the pain of the community in an effort to cause individual change?
Do we use laws and policy to constrain the use of algorithms for specific purposes, in an effort to regulate the effects away?
Do we build alternate tools with the same functionality and take users away from the harm-causing tools?
Do we use our power as laborers to strike against the harm caused by the tools that we build?

With climate change (and too with data security and privacy), we’re already taking all of these approaches. What else might be out there? What else can we do to lead to change?

Repersonalizing Digital Communications: Against Standardizing and Interfering Mediations

Wed, 12 Aug 2020 10:36:17 +0000

Back in 2013 I wrote a blog post reacting to Cristina Vanko’s project to handwrite her text messages for one week. At the time, I focused on how Cristina introduced slowness into a digital communication that often operates as a conversation due to the immediacy and frequency of responses. Since 2013, texting has grown more popular and instant messaging has woven its way into our work environments as well. Reinvoking that slowness stays relevant, but careful notification settings can help recapture it as well.

What I want to focus on is the way that her project repersonalizes the digital medium of communication, adding her handwriting and therefore more of her personality into the messages that she sends. I thought of this project again while watching a talk from Jonathan Zong for the Before and Beyond Typography Online Conference. In his talk, he points out that “writing is a form of identity representation”, with handwriting being “highly individualized and expressive”, while “in contrast, digital writing makes everyone’s writing look the same. People’s communications are filtered through the standardized letterforms of a font.”

His project that he discusses in part of that talk, Biometric Sans, “elongates letterforms in response to the typing speed of the individual”, thus providing another way to reembody personality into digitally-mediated communications. He describes the font as “a gesture toward the reembodiment of typography, the reintroduction of the hand in digital writing.” It’s an explicit repersonalization of a digitally-mediated communication, in much the same way Cristina Vanko chose to handwrite her text messages to do the same. Both projects seek to repersonalize, and thereby rehumanize, the somewhat coldly standardized digital communication formats that we rely on.

Without resorting to larger projects, we find other ways to repersonalize our digital communications: sharing stickers (I’m rather fond of Rejoinders), crafting new expressions (lol) and words, and even sending voice responses (at times accidentally) in text messages. In this way we can poke at the boundaries of the digital communication methods sanitized by standardized fonts for all users.

While Jonathan stayed rather focused on the typography mediation of digital communication due to the topic of the conference, I want to expand this notion of repersonalizing the digital communication methods. Fonts are not the only mechanism by which digital communications can be mediated and standardized—the tools that we use to create the text displayed by the fonts do just as much (if not more).

The tools that mediate and standardize our text in other ways are, of course, automatic correction, predictive text, and the software keyboards themselves.

Apple is frustratingly subtle about automatic correction (autocorrect), oftentimes changing a perfectly legitimate word that you’ve typed into a word with a completely different meaning. It’s likely that autocorrect is attempting to “accelerate” your communications by guessing what you’re trying to type. This guess, mediating your input to alter the output, often interferes with your desired meaning. When this interfering mediation fails (which is often), you’re instead slowed down, forced to identify that your intended input has been unintentionally transformed, fix it, perhaps fix it again, and only then send your message.

Google, meanwhile, more often preemptively mediates your text. Predictive text in Google Mail “helps” you by suggesting commonly-typed words or responses.

This is another form of interference (in my mind), distracting you from what you’re actually trying to communicate and instead inserting you into a conflict with the software, fighting a standardized communication suggestion while you seek to express your point (and your personality) with a clear communication. Often, it can be distractingly bland or comical.

In Google Mail, this focus on standardized predictive responses also further perpetuates the notion of email as a “task to be completed” rather than an opportunity to interact, communicate, or share something of yourself with someone else.

Software keyboards themselves also serve to mediate and effectively standardize digital communications. For me personally, I dislike software keyboards because I’m unable to touchtype on them (Frustrated, I tweeted about this in January). Lacking any hardware feedback or orientation, I frequently have to stare at the keyboard while I’m typing. I’m less able to focus on what I’m trying to say because I’m busy focusing on how to literally type it. This forced slowness, introducing a max speed at which you can communicate your thoughts, effectively forces you to rely on software-enabled shortcuts such as autocorrect, predictive text, or actual programmed shortcuts (such as replacing “omw” with “On my way!”), rather than being able to write or type at the speed of your thoughts (or close to it). Because of this limitation, I often choose to write out more abstract considerations or ideas longhand, or reluctantly open my computer, so that I have the privilege of a direct input-to-output translation without any or extensive software mediation.

In a talk last June at the SF Public Library, Tom Mullaney discussed the mediation of software keyboards in depth, pointing out that software keyboards (or IMEs as he referred to them) do not serve as mechanical interpreters of what we type, but rather use input methods to transcribe text, and that those input methods can adapt to be more efficient. He used the term “hypography” to talk about the practice of writing when your input does not directly match the output. For example, when you use a programmed shortcut like omw, but also when you seek to type a character that isn’t represented on a key, such as ö, or if you’re typing in a language that uses a non-latin alphabet, a specific sequence of keystrokes to represent a fully-formed character in written text. Your input maps to an output, rather than the output matching the input.

These inputs are often standardized, allowing you to learn the shortcuts over time and serving the purpose of accelerating your communications, but in the case of autocorrect or predictive text, they’re frequently suffering from new iterations—new words or phrases that interferingly mediate and change a slip up into a skip up, encourage you to respond to an email with a bland “Great, thanks!” or attempt to anticipate the entire rest of your sentence after you’ve only written a few words. Because I also have a German keyboard configured, my predictive text will occasionally “correct” an English typo into a German word, or overcapitalize generic English nouns by mistakenly applying German language rules.

All of these interfering and distracting mediations that accelerate and decelerate our digital communications, alongside our ongoing efforts to repersonalize those communications, has me wondering: What do we lose when our digital communications are accelerated by expectations of instantaneous responses? What do we lose when they’re decelerated by interfering mediations of autocorrect? What do we lose when our communications are standardized by fonts, predictive text, and suggested responses?

Listening to Music while Sheltering in Place

Mon, 25 May 2020 10:08:31 +0000

The world is, to varying degrees, sheltering-in-place during this global coronavirus pandemic. Starting in March, the pandemic started to affect me personally:

I started working from home on March 6th.
Governor Gavin Newsom announced on March 11 that any gatherings over 250 people were strongly discouraged, effectively cancelling all concerts for the month of March.
On March 16th, the mayor of San Francisco along with several other counties in the area, announced a shelter-in-place order.

Ever since then, I’ve been at home. Given all these changes in my life, I was curious what new patterns I might see in my music listening habits.

With large gatherings prohibited, I went to my last concert on March 7th. With gatherings increasingly cancelled nationwide, and touring musicians postponing and cancelling events, March 27th, Beatport hosted the first livestream festival, “ReConnect. A Global Music Series”. Many more followed.

Industry-wide studies and data analysis have attempted to unpack various trends in the pandemic’s influence on the music industry. Analytics startup Chartmetric is digging into genre-based listening, geographical listening habits, and Billboard and Nielsen conducting a periodic entertainment tracker survey.

Because I’m me, and I have so much data about my music listening patterns, I wanted to explore what trends might be emerging in my personal habits. I analyzed the months March, April, and May during 2020, and in some cases compared that period against the same period in 2019, 2018, and 2017. The screenshots of data visualizations in this blog post represent data points from May 15th, so it is an incomplete analysis and comparison, given that May in 2020 is not yet complete.

Looking at my listening habits during this time period, with key dates highlighted, it’s clear that the very beginning of the crisis didn’t have much of an effect on my listening behavior. However, after the shelter-in-place order, the amount of time I spent listening to music increased. After that increase it’s remained fairly steady.

Key dates such as the first case in the United States, the first case in California, and the first case in the Bay Area are highlighted along with other pandemic-relevant dates.

Listening behavior during March, April, and May over time

When I started my analysis, I looked at my basic listening count from traditional music listening sources. I use Last.fm to scrobble my listening behavior in iTunes, Spotify, and the web from sites like YouTube, SoundCloud, Bandcamp, Hype Machine, and more.

If you just look at 2018 to 2020, it seems like my listening habits are trending upward, maybe with a culmination in 2020. But comparing against 2017, it isn’t much of a difference. I listened to 25% fewer tracks in 2018 compared with 2017, 19% more tracks in 2019 compared with 2018, and 25% more tracks in 2020 compared with 2019.

If I break that down by when I was listening by comparing my weekend and weekday listening habits from the previous 3 years to now, there’s still perhaps a bit of an increase, but nothing much.

With just the data points from Last.fm, there aren’t really any notable patterns. But number of tracks listened to on Spotify, SoundCloud, YouTube, or iTunes provides an incomplete perspective of my listening habits. If I expand the data I’m analyzing to include other types of listening—concerts attended and livestreams watched—and change the data point that I’m analyzing to the amount of time that I spend listening, instead of the number of tracks that I’ve listened to, it gets a bit more interesting.

While the number of tracks I listened to from 2019 to 2020 increased only 25%, the amount of time I spent listening to music increased by 74%, a full 150 hours more than the previous year during this time period. And May isn’t even over yet!

It’s worth briefly noting that I’m estimating, rather than directly calculating, the amount of time spent listening to music tracks and attending live music events. To make this calculation, I’m using an estimate of 3 hours for each concert attended, 4 hours for each DJ set attended, 8 hours for each festival attended, and an estimate of 4 minutes for each track listened to, based on the average of all the tracks I’ve purchased over the past two years. Livestreamed sets are easier to track, but some of those are estimates as well because I didn’t start keeping track until the end of April.

I spent an extra 150 hours listening to music this year during this time—but when was I spending this time listening? If I break down the amount of time I spent listening by weekend compared with weekdays, it’s obvious:

Before shelter-in-place, I’d spend most of my weekends outside, hanging out with friends, or attending concerts, DJ sets, and the occasional day party. Now that I’m spending my weekends largely inside and at home, coupled with the number of livestreaming festivals, I’m spending much more of that time listening to music.

I was curious if perhaps working from home might reveal new weekday listening habits too, but the pattern remains fairly consistent. I also haven’t worked from home for an extended period before, so I don’t have a baseline to compare it with.

It’s clear that weekends are when I’m doing most of my new listening, and that this new listening likely isn’t coming from my traditional listening habits. If I split the amount of time that I spend listening to music by the type of listening that I’m doing, the source of the added time spent listening is clear.

Hello, livestreams. If you look closely you can also spy the sliver of a concert that I attended on March 7th.

Livestreams dominate, and so does Shazam

All of the livestreams I’ve been watching have primarily been DJ sets. Ordinarily, when I’m at a DJ set, I spend a good amount of time Shazamming the tracks I’m hearing. I want to identify the tracks that I’m enjoying so much on the dancefloor so I can track them down, buy them, and dig into the back catalog of those artists.

So I requested my Shazam data to see what’s happening now that I’m home, with unlimited, shameless, and convenient access to Shazam. For the time period that I have Shazam data for, the correlation of Shazam activity to number of livestreams watched is fairly consistent at roughly 10 successful Shazams per livestream.

Given the correlation of Shazam data, as well as the continued focus on watching DJ sets, I wanted to explore my artist discovery statistics as well. Especially when it seemed like my listening activity hadn’t shifted much, I was betting that my artist discovery statistics have been increasing during this time. If I look at just the past few years, there seems to be a direct increase during this time period.

However, after I add 2017 into the list as well, the pattern doesn’t seem like much of a pattern at all. Perhaps by the end of May, there will be a correlation or an outsized increase. But at least for now, the added number of livestreams I’ve been watching don’t seem to be producing an equivalently high number of artist discoveries, even though they’re elevated compared with the last two years.

That could also be that the artists I’m discovering in the livestreams haven’t yet had a substantial effect on my non-livestream listening patterns, even if there’s 91 hours of music (and counting) in my quarandjed playlist where I store the tracks that catch my ear in a quarantine DJ set. Adding music to a playlist, of course, is not the same thing as listening to it.

Livestreaming as concert replacement?

Shelter-in-place brought with it a slew of event cancellations and postponements. My live events calendar was severely affected. As of now, 15 concerts were affected in the following ways:

The amount of time that I spend at concerts compared with watching livestreams is also starkly different.

I’ve spent 151 hours (and counting) watching livestreams, the rough equivalent of 50 concerts—my entire concert attendance of last year. This is almost certainly because I’m often listening to livestreams, rather than watching them happen.

Concerts require dedication—a period of time where you can’t really do anything else, a monetary investment, and travel to and from the show. Livestreams don’t have any of that, save a voluntary donation. That makes it easier to turn on a stream while I’m doing other things. While listening to a livestream, I often avoid engaging with the streaming experience. Unless the chat is a cozy few hundred folks at most, it’s a tire fire of trolls and not a pleasant experience. That, coupled with the fact that sitting on my couch watching a screen is inherently less engaging than standing in a club with music and people surrounding me, means that I’m often multitasking while livestreams are happening.

The attraction for me is that these streams are live, and they’re an event to tune into, and if you don’t, you might miss it. Because it’s live, you have the opportunity to create a shared collective experience. The chatrooms that accompany live video streams on YouTube, Twitch, and especially with Facebook’s Watch Party feature for Facebook Live videos, are what foster this shared experience. For me, it’s about that experience, so much so that I started a chat thread for Jamie xx’s 2020 Essential Mix so that my friends and I could experience and react to the set live.

This personal experience is contrary to the conclusion drawn in this article on Hypebot called Our Music Consumption Habits Are Changing, But Will They Remain That Way? by Bobby Owsinski: “Given the choice, people would rather watch something than just listen.”.

Given the choice, I’d rather have a shared collective experience with music rather than just sit alone on my couch and listen to it.

Of course, with shelter-in-place, I haven’t been given a choice between attending concerts and watching livestreamed shows. It’s clear that without a choice, I’ll take whatever approximation of live music I can find.

Why the quality of audio analysis metadatasets matters for music

Mon, 16 Mar 2020 10:51:26 +0000

I’ve been thinking for some time about the derived metadata that Spotify and other digital streaming services construct from the music on their platforms. Spotify’s current business revolves around providing online streaming access to music and podcasts, as well as related content like playlists, to users.

Like any good SaaS business, their primary goal is to acquire and keep customers. As a digital streaming service business, the intertwined goal is to provide quality content to those customers. The best way to do both of those is to derive and collect metadata about customer usage patterns, but also about the content being delivered to the customers. The more you know about the content being delivered, the more you can create new distribution mechanisms for the content and make informed deals to acquire new content.

Creating metadatasets from the intellectual property of artists

Today, when labels and distributors provide music to digital streaming services (artists can’t provide it directly), they grant those services permission to make the music tracks available to users of the digital streaming services. Based on my review of the Spotify Terms and Conditions of Use, Spotify for Artists Terms and Conditions, and the distribution agreement for a commonly-used distribution service, DistroKid, artists don’t grant explicit permission for what services do next—create metadata about those tracks. An exception relevant with the DistroKid distribution agreement is if artists sign up for an additional service, DistroLock, they then are bound by an additional addendum granting the service permission to create an audio fingerprint to uniquely represent the track so that it can be used for copyright enforcement and possibly to pay out royalties.

In his book Metadata, Jeffrey Pomerantz defines metadata as “a means by which the complexity of an object is represented in a simpler form.” In this case, streaming services like Spotify create different types of metadata to represent the complexity of music with various audio features, audio analysis statistics, and audio fingerprints. The services also gather “use metadata” about how customers use their services—at what point in a song a person hits skip, what devices they use to listen, their location when listening, and other data points.

Creating metadatasets is crucial to delivering content

Pandora has patents for the types of music metadata that they create, that behind the “music genome project”. Spotify also has patents (and a crucial one from their acquisition of the Echo Nest) to do the same, as well as many that cover the various applications of those metadata.

These companies can use these metadatasets as marketing tools, as we’ve seen with the #SpotifyWrapped campaign; to correlate the music metadata with use metadata, such as to create new music marketing methods like contextual playlists; to select advertising that matches up sonically well with the tracks being listened to; and to provide these insights to artists and labels, making them more reliant on their service as a distribution and marketing mechanism.

Spotify currently provides a subset of the insights they derive from the combination of use metadata with music track metadata to artists with the Spotify for Artists service. The end user license agreement for the service makes it clear that it’s a free service and Spotify cannot be held responsible for the relative accuracy of the data available. Emphasis mine:

Spotify for Artists is a free service that we are providing to you for use at our discretion. Spotify for Artists may provide you with the ability to view demographic data on your fans and usage data of your music. While we work hard to ensure the accuracy of the data, we do not guarantee that the Spotify for Artists Service or the data that we collect from the Service will be error-free or that mistakes, including mistakes in the data insights that we provide to you, will not happen from time to time.

Spotify for Artists Terms and Conditions

It’s likely that some labels have already negotiated access to various insights and metadata that Spotify creates and collects.

Other valuable insights that can be derived from these metadatasets include: the types of music that people listen to in certain cities, which tracks are most popular in certain cities, what types of music people tend to listen to in different seasons, and even what types of music people of different ages, genders, education levels, and classes tend to listen to.

These insights, provided to artists, labels, and distributors, guide marketing campaigns, tour planning, artist-specific investments, and even music production styles. Thing is, it’s tough to decipher exactly how these companies create the metadatasets that all these valuable insights rely on, and how the accuracy of that metadata is (if at all) validated.

How the metadatasets get made

In an episode of Vox Earworm, the journalist Matt Daniels of The Pudding and Estelle Caswell of Vox briefly discuss how the metadatasets of Spotify and Pandora were created, pointing out that Spotify has 35 million songs, but the metadataset is algorithmically generated. Meanwhile, Pandora has only 2 million songs, but those 450 total attributes were defined and applied by a combination of trained musicologists and algorithms to the songs. Their discussion starts at 1:45 in this episode and continues for about 90 seconds.

The features in the metadatasets have been defined by algorithms written by trained musicologists, amateur musicians, or even ordinary data scientists without musical training or expertise. The specific features collected by Spotify are publicly available in their audio features API and audio analysis API endpoints, and both include metadata that objectively describe each track, such as duration, as well as more subjective features such as acousticness, liveness, valence, and instrumentalness.

The more detailed audio analysis API features splits up each track into various sections and segments, and computes features and confidence levels for each of the sections and segments.

Spotify, building off the Echo Nest technology, relies on web scraping and algorithms to create these metadatasets. According to a patent filed by the Echo Nest in 2011, three different types of metadata are created:

Acoustic metadata, which is the “numerical or mathematical representation of the sound of a track”,
Cultural metadata, which “refers to text-based information describing listener’s reactions to a track or song”, and
Explicit metadata, which “refers to factual or explicit information relating to music”.

The explicit metadata is information such as “track name” or “artist name” or “composer, while the acoustic metadata can be an acoustic fingerprint to represent the song, or can include features like “tempo, rhythm, beats, tatums, or structure, and spectral information such as melody, pitch, harmony, or timbre.” The cultural metadata is where the more subjective features come from, and it can come from a variety of different subjective sources: “expert opinion such as music reviews”, “listeners through Web sites, chat rooms, blogs, surveys, and the like”, as well as information “generated by a community of listeners and automatically retrieved from Internet sites, chat rooms, blogs, and the like.” The patent gives other examples such as “sales data, shared collections, lists of favorite songs, and any text information that may be used to describe, rank, or interpret music.” It can also build off of existing databases made available by companies like Gracenote, AllMusic (referenced as AMG, now RhythmOne, in the patent), and others.

Pandora doesn’t share an API for their Music Genome Project data, but they do mention that it contains 450 total attributes, or features in the data. I dug into their patents and it is clear that the number of features used varies depending on the type of music, and the features given as examples in the patents range from vocalist gender, distortion in electric guitar, type of background vocals, genre, era, syncopation, and lead vocal present in song(also). Pandora uses a combination of musicologists and algorithms to assign values.

Representation in the metadatasets, representation in the taco bell

We know a little about how Spotify and Pandora create their metadatasets. We know less about how representative those metadatasets are, both in terms of feature coverage and music coverage.

Barely knowing which features are available for Pandora, and even while having a decent idea of what Spotify has available, it’s possible that the features that exist in the metadatasets are incomplete. The features in the metadatasets could be limited to those that were the easiest to compute at the time, those that are deemed interesting by the creators, or even those that are highly-correlated with profitable user behavior. It’s expensive to create, store, and apply new metadata features, so businesses must have a clear value proposition before developing new models or tasking more musicologists with the creation of a unique audio feature.

Based on the locations of Spotify, Pandora, and the companies informing their metadatasets, it’s likely that the datasets that these metadatasets and their features are built on aren’t representative of music worldwide but instead include bias toward music that is easily available in their geographic locations.

The size of the datasets that underpin the metadata creation varies—Pandora has 2 million tracks, Spotify has 35 million—the representativeness of the data sample is more important than the size. And that is a variable that we have almost no information about.

I haven’t done (and can’t do) the data analysis to determine the distribution of tracks in those giant datasets. Without that I can only speculate:

It’s possible that both of them have a disproportionate concentration of artists that create and record music in the United States and Western Europe.
It’s almost certain that both of those datasets contain only music recorded in the digital or digital-adjacent eras. Music recorded in analog tape eras that haven’t been digitized can’t be represented in the datasets.
It’s unlikely that the datasets include music by artists lacking the internet connection necessary to digitally distribute their music, even if it is digitized.

We could learn more about the representativeness of the datasets used to create the metadatasets if we knew more about how the metadatasets themselves are validated. But again, that’s another area that lacks clarity.

How the metadatasets get validated… or not

Their uniqueness of their businesses are built on these metadatasets, but it doesn’t seem like there are processes in place to validate the features developed and in use by Pandora and Spotify across the industry. There’s no central database of tracks that I know of, a “Tom’s Diner” of audio feature validation, that can be used to tune the accuracy of audio features that exist in multiple industry metadatasets. Instead, much like the lossy compression of an MP3, there is just the “close enough for our purposes” approximation for validation.

Pandora uses its musicologists to validate the features assigned to tracks by other musicologists and by algorithms, and uses a selection and ranking module to arrive at a “wisdom of the crowd of experts” result for the eventual list of features associated with a track. The accuracy of a feature is a relative score based on how many other experts associated that same feature with a track.

Spotify uses a prediction model to predict the subjective (and harder-to-compute) features such as liveness, valence, danceability, and presence of spoken word lyrics. In the patent filing, they disclose the validation methods used for the features predicted by that model:

Comparing the results of the model to a “ground truth dataset” created from already-labeled data sourced in part from “crowdsourced online music datasets such as SOUNDCLOUD, LAST.FM, and the like” [sic].
Evaluating the percentage of true positives, false negatives, and true negatives returned by the model predictions for features with a binary value (true or false).

The patent then describes taking appropriate steps to bolster training data and improve coverage of the datasets to produce more accurate results in response to the validation results. However, since this is a patent filing rather than a blog post describing their data science practices, we don’t know how often the prediction models and training datasets are updated, or what other methods are used to compile and validate the training datasets themselves.

Lacking an objectively true value for many of these audio features, it’s difficult for services to reliably validate their metadatasets. In fact, rather than comparatively validating their metadatasets, many of the metadatasets are built on top of each other. The Spotify patent for the prediction model makes it clear that the “ground truth dataset” used for validation is partially sourced from other metadatasets. This Echo Nest patent that I discussed earlier makes it clear that different types of metadata can come from pre-existing metadatasets.

Without large-scale understanding of metadata validity across these existing metadatasets, it’s likely that errors and biases in the metadata can proliferate as new ones are created. Eventually, that lack of quality metadata can have a disproportionate effect on the artists creating the music that this metadata is derived from.

Why metadata quality matters

Spotify and Pandora both rely extensively on these metadatasets to deliver valuable streaming services to customers and to create engaging content like playlists and stations for their listeners. Spotify has positioned itself as a valuable distribution and marketing mechanism for artists, to the point that they’ve devised a new scheme where artists and labels can pay for privileges like prominent playlist placement or spotlights in Spotify.

Metadata underpins the business model of these companies, shaping our experience of music by directly affecting how music is distributed and consumed. But we don’t know how valid the metadata is, we don’t know if it’s biased, and we don’t know how much of a feedback loop is involved in its interpretation to create new distribution and consumption mechanisms.

If these companies don’t do more to improve the quality of metadata, artists can lose revenue and miss out on distribution opportunities. Listeners can get bored by the sameness of playlists, or the inaccurate interpretations of their radio station requests, and stop using Spotify and Pandora to discover new music. Without representative and valid metadata, music loses.

What went into writing this

I read a lot over the past few months that informed my thinking in this essay, or some of the points that I made, without being something I quoted or linked directly in the text. I also am grateful to the conversations I had with my former colleague Jessica about this topic, and the feedback that my former colleague Neal gave me on an earlier version of this post.

Spotify background

I read the Spotify API documentation for the audio features and audio analysis endpoints.
- Get Audio Analysis for a Track
- Get Audio Features for a Track
The Spotify for Artists FAQ was informative, especially the following questions.
Posts on the Spotify engineering blog, Spotify Labs.
Patents filed by Spotify or The Echo Nest, in an attempt to learn how they create music metadata:
- US8280889B2 - Automatically acquiring acoustic information about music
- US10089578B2 - Automatic prediction of acoustic attributes from an audio signal
A Hacker Noon article by Sophia Ciocca: Spotify’s Discover Weekly: How machine learning finds your new music
This Hypebot article by Bruce Houghton: Spotify’s Paid Promotion Tool Is Called Marquee and Artists, Indie Labels Can’t Afford To Use It

Pandora background

An article in the New York Times Magazine by Rob Walker: The Song Decoders at Pandora
A sponsored article in Forbes Insights by their Insights Team: Forbes Insights: How Pandora Knows What You Want To Hear Next
Two essays in the East Bay Express:
- By Chris Parker: Personal Shoppers
- By Kara Platoni: Pandora’s Box
Several Pandora patents in an attempt to learn about some of the features that they create and how they create them:

What it takes to get to a concert

Sun, 09 Feb 2020 11:26:05 +0000

Ticket buying in the modern era is pretty brutal. You find out your favorite artist is coming to town, and with any luck, you discover this before the tickets go on sale. Then you start planning to get tickets. Set up a calendar reminder with a link to the site, then you get ready. If there are presales, you ask friends or you check emails — if you’re a dedicated concertgoer, you probably get emails from the promoters, venues, and maybe even your favorite artists’ fan clubs — tracking down the codes.

Then you get ready, mouse pointer cued up at 9:59, waiting until tickets go on sale. The time flips, it’s 10:00 AM and you click! Prepared to quickly select 2, best available (or GA floor, because who wants a balcony seat), and add to cart. But wait! You see the dreaded message. You’re in a queue. Now all you can do is desperately stare at the webpage, hoping nothing changes. What if a browser extension interferes? What if your browser freezes up? Finally, you’re out of the queue. You go to select your tickets, but wait. GA is all gone. All that’s left is the seated Loge. For a band that you dance to. Or worse, it’s already sold out. All that time, all that anxiety, all that preparation, only to get shut down.

And that’s just the presale. You’ll do the whole thing over again at the next presale, or during the general onsale, hoping that the artist and the venue were strategic enough to set some tickets aside for each sale. If it comes down to it, you might have to show up to the venue an hour early (or more) before the show starts to get one of the limited tickets available at the door.

That’s everything a dedicated concertgoer goes through to get concert tickets. Thing is, according to Kaitlyn Tiffany in The Atlantic, that’s also what modern ticket scalpers do.

This week was a brutal one for ticket sales for me and my friends. A show at a 2000+ capacity venue sold out within a few minutes during the presale, and a second show added later also sold out within minutes. The Format announced their first live dates in years, playing 2 shows in NYC and Chicago both, and 1 show in Phoenix. The presale tickets for all the shows sold out within a minute, or in the case of Phoenix, was plagued by ticket website issues but still managed to sell out by the end of the day. By the time the general ticket sales happened, they’d announced an additional show in each city. The general ticket sales also sold out within minutes, and Phoenix ended up with a third show before the day was up.

How does it happen? And why do we put ourselves through this?!

It’s important to note that buying concert tickets at all is a privilege. Some people (like me) make it a lifestyle to go to concerts and DJ sets. Others save their money and spend big to get great seats to see favorite artists in arena shows. But it takes money, time, and a bit of luck (or planning) to get tickets and get to a show.

Whether or not you manage to get tickets to a show depends on several factors:

Did you hear about the show before the tickets went on sale?
Did you have enough money at the time tickets went on sale (and in general) to afford the tickets?
Is your work schedule stable enough to know that you can go to the show if you buy tickets immediately when they go on sale?

If any one of these factors doesn’t work out, then you don’t have tickets to the show. Whether or not you get the opportunity to see an artist perform in concert at all is up to a whole other set of factors, subject to the careful strategies of the music industry combined with the artistic whims of the performers.

If an artist doesn’t have a big enough fanbase in your city, and if it isn’t geographically convenient with available music venues, the artist probably won’t stop in your city. Even if they stop, the venue size can play a crucial role in whether or not you’ll get tickets to the show—will they be available, and will you even want them?

Artists, especially after they’ve “gotten big”, can crave smaller, more intimate shows. But those are the shows that tend to sell out in a minute—especially if the fanbase in a certain city is larger than anticipated or if the artist is only playing a limited number of shows and end up drawing people from out of the ordinary reach of a venue.

Other times, artists can analyze the size of their fanbase in a city and then choose a venue—without considering if the venue size is appropriate for their type of music. Bon Iver toured 20,000+ seat arenas on their last tour, while they’re famous for their intimate music and have videos on YouTube with hundreds of thousands of views of Justin Vernon playing to just 1 fan. Even if an artist’s fanbase is large enough to fill an arena, the fans still might not want to buy tickets to see them in an arena.

Beyond those considerations, artists can’t always play the venues they want to play due to promoter restrictions or other industry partnerships, sometimes leading to uncharacteristic bookings at oddly-sized or oddly-shaped venues: DJs playing a concert hall, rock bands in a semi-seated venue, or possibly even skipping a city entirely.

The venue an artist chooses (or is forced to choose) can be a key factor when you’re deciding if you want to get tickets. But the artist (and their tour manager, and others) have still more to do before this concert happens.

The ticket prices have to be set. Surely venues and promoters have set costs and prices that end up as effective ticket minimums for many shows, but artists certainly have a level of influence as well. Especially high-profile artists like Taylor Swift have chosen on past tours to make affordable tickets available to their fans.

And therein lies the rub: artists can price competitively, or highly, knowing they can charge a certain price and still sell out their show (or nearly sell it out). But they can also price affordably, hoping that legitimate fans will be able to snap up tickets when they go on sale, rather than delaying their purchase and being forced to buy from scalpers.

OK so we’re still trying to buy these concert tickets. You’ve heard about the show, the artist has booked the venue and priced their tickets, you’ve got the money, you’ve got the time, you are ready at 10am on a Friday (or a Wednesday or a Thursday for those sweet sweet presale tickets). Where are you buying your tickets?

Ticket sites range from the homegrown (see: Bottom of the Hill), new kids on the block (Big Neon, Tixr), the budding behemoths (Eventbrite, AXS, Etix) and the (despised) old guard (TicketWeb/Ticketmaster/LiveNation). If you’re rushing to buy online tickets, you also need to prepare for the site experience.

If it isn’t a site you’ve used before, you might want to consider if it requires an account to buy tickets. If it does, you have to make one and make sure you’re signed in before you try to buy the tickets. You also want to consider if the show big enough that you’ll end up in a queue to buy the tickets, and if the site is reliable enough to handle the load of a lot of people trying to buy tickets without crashing or throwing an error.

Beyond site reliability, you have to consider your personal threshold for every ticket-buyer’s worst nightmare: fees. Almost every ticket purchase includes fees. How high do the fees need to be before you abandon your ticket purchase entirely? You also have to consider if there will be fees added to the face value of the ticket, and how high are too high of fees before you abandon the ticket purchase entirely. Of course, the irony of paying ticket fees is that most fans (myself included) dislike paying them because for so long the fees are hidden—last minute additions to your total, spiking the cost of $35 tickets to $60 at times. But it can be argued that transparently-disclosed fees are acceptable, and even necessary to provide a resilient, secure, reliable ticketing site—as well as to pay the promoters working hard to make sure your favorite band actually stops in your city.

Artists, promoters, venues, and ticketing sites do a lot to try to prevent ticket scalpers from bombing the market and selling out a show in minutes only to relist the tickets minutes later at unbelievable prices. Innovations in ticket technology, new marketplaces, and just plain making it harder to get tickets:

Dave Chappelle announced on a Friday that he was playing 1 Saturday and 2 Sunday shows in San Francisco, with the tickets going on sale on the Saturday of the first show.
Ticketmaster added the ability for artists or venues to require a rotating barcode for mobile tickets, preventing screenshots of tickets from being sold. Unfortunately, this was enforced with disastrous effect at a Black Keys show in LA. Anti-reselling measures only work if they punish the resellers, instead of the buyers, which is what happened in this case.
Nine Inch Nails required fans purchase tickets in person to go to their San Francisco show in 2018. Which is great, for all the people that can afford to take time off and stand in a line.
Pearl Jam worked with Ticketmaster to set up a fan-to-fan face value ticket exchange to control the reselling market for their tour.
Artists also commonly say that they’re working with venues and ticket sites to identify legitimate ticket purchasers compared to scalpers.

What makes a ticket purchaser legitimate? Probably some degree of purchasing tickets in a specific geographic region and in clusters of genres, likely combined with some fraud analysis. Then I wonder how suspicious my own ticket purchasing habits must look to the algorithms at times. As long as we’re attempting to define what a legitimate ticket purchaser looks like, we can consider who deserves the presale codes for shows.

There’s a notion that only “real fans” deserve first access to presale codes and tickets. But how do you verify and validate true fans? You could use specific digital consumption patterns, such as those that are probably used to give out Spotify presale codes, but those are limited to only those listening habits that are directly observable in digital data. Artists want people to buy tickets to their shows—that’s why often, presale codes are straightforward to track down.

Most often, getting tickets to a show is a matter of knowing the right people at the right time that might have information you don’t have. Songkick is there to fill in the gaps, alongside emails and texts from promoters and venues. But ultimately, nothing beats having a community of fans. And that was the thing that fascinated me about the article in The Atlantic about the modern ticket scalpers. Me and my friends, we use many of the same tactics to buy tickets. It’s a privilege and a challenge to get the tickets we want, but we love going to concerts. And often, it feels like it’s the only way these days we can help artists make money.

Problems with Indexing Datasets like Web Pages

Tue, 28 Jan 2020 12:55:03 +0000

Google has created a dataset search for researchers or the average person looking for datasets. On the one hand, this is a cool idea. Datasets are hard to find in cases, and this ostensibly makes the datasets and accompanying research easier to find.

In my opinion this dataset search is problematic for two main reasons.

1. Positioning Google as a one-stop-shop for research is risky.

There’s consistent evidence that many people (especially college students who don’t work with their library) start and end their research with Google, rather than using scholarly databases, limiting the potential quality of their research. (There’s also something to be said here about the limiting of access to quality research behind exploitative and exclusionary paywalls, but that’s for another discussion).

Google’s business goal of being the first and last stop for information hunts makes sense for them as a company. But such a goal doesn’t necessarily improve academic research, or the knowledge that people derive based on information returned from search results.

2. Datasets without datasheets easily lead to bias.

The dataset search is clearly focused on indexing and making more available as many datasets as possible. The cost of that is continuing sloppy data analysis and research due to the lack of standardized Datasheets for Datasets (for example) that fully expose the contents and limitations of datasets.

The existing information about these datasets is constructed based on the schema defined by the dataset author, or perhaps more specifically, the site hosting the dataset. It’s encouraging that datasets have dates associated with them, but I’m curious where the description for the datasets are coming from.

Only the description and the name fields for the dataset are required before a dataset appears in the search. As such, the dataset search has limitations. Is the description for a given dataset any higher quality than the Knowledge Panels that show up in some Google search results? How can we as users independently validate the accuracy of the dataset schema information?

The quality of and details provided in the description field vary widely across various datasets (I did a cursory scan of datasets resulting from a keyword search for “cheese”) indicating that having a plain text required field doesn’t do much to assure quality and valuable information.

When datasets are easier to find, that can lead to better data insights for data analysts. However, it can just as easily lead to off-base analyses if someone misuses data that they found based on a keyword search, either intentionally or, more likely, because they don’t fully understand the limitations of a dataset.

Some vital limitations to understand when selecting one for use in data analysis are things like:

What does the data cover?
Who collected the data?
For what purpose was the data collected?
What features exist in the data?
Which fields were collected and which were derived?
If fields were derived, how were they derived?
What assumptions were made when collecting the data?

Without these valuable limitations being made as visible as the datasets themselves, I struggle to feel overly encouraged by this dataset search in its current form.

Ultimately, making information more easily accessible while removing or obscuring indicators that can help researchers assess the quality of the information is risky and creates new burdens for researchers.

Wrapping up the year and the decade in music: Spotify vs my data

Thu, 05 Dec 2019 15:01:40 +0000

Spotify’s 2019 Wrapped aims to give you an overview of your past year’s listening habits. It proclaims: these were your top 5 tracks and artists! You spent this much time listening to your favorite artist!

This year (the last year of the decade) they also expanded to all of the 2010s, sharing the top artists and tracks for each year in the decade that you used Spotify.

Because I have my own data that combines Last.fm listening data, my iTunes music library, and concert-relevant activities, this is my comparison of Spotify’s data with my own listening habits (more exhaustively tracked).

I have Last.fm set up to monitor Spotify, but also tracks that I listen to in Google Chrome, using the Music app on my iPhone, and local iTunes listening on my personal laptop. Spotify, of course, just sees Spotify.

According to Spotify, my top 5 artists were:

Tourist
Manatee Commune
Lane 8
Amtrac
SebastiAn

According to my own data, my top 5 artists were:

Tourist
Lane 8
Benoit & Sergio
The Vaccines
Litany

Manatee Commune was just 4 listens behind Litany, with 76 total listens for the year so far. It’s an impressive showing from them, considering that I never ended up purchasing any tracks by them. I own full albums or several tracks by all the other artists in both of my top 5 lists, making it easier for me to rack up listens—I listen only to music that I own or untrackable DJ sets in SoundCloud while I’m mobile.

The track stats are where my data really starts to differ from Spotify’s…

My top 5 tracks according to Spotify are:

Wow that’s a lot of Manatee Commune! Let’s see how the listens of those tracks stack up:

Manatee Commune - W/O | 10 listens
Odesza - Just A Memory (Mild Minds Remix) | 7 listens
Aloe Blacc - Brooklyn in the Summer | 8 listens
Manatee Commune - What We’ve Got (feat. Flint Eastwood) | 11 listens
Manatee Commune - Raspberry Puree | 10 listens

What were my “actual” top 5 songs of 2019?

I’ll cheat and go to my top 7 because my first 2 top 5 songs just prove that I struggled to sleep a lot and listened to my Insomnia playlist…

Hey Rosetta! - Trish’s Song | 28 listens
The Cinematic Orchestra - That Home Extended | 26 listens
The Chemical Brothers - Got To Keep On | 25 listens
Justin Jay - I’m Shy When I’m Around You | 23 listens
Benoit & Sergio - The Way You Get | 22 listens
Poolside - Everything Goes (Body Music Remix) | 21 listens
Tourist - Elixir | 20 listens

Pretty stark difference in those lists and those numbers. Manatee Commune is nowhere in sight. A large reason for that is because my listening pattern with Manatee Commune almost perfectly lines up with seeing them live (indicated by the orange triangle and dotted line):

But enough about Manatee Commune. Let’s talk about the real star of 2019: Tourist! All of the data agrees that he was my top artist of 2019. I saw him twice in concert (and I’ll see him again in a couple weeks).

I’ve listened to him sporadically since 2012, first listening to a track of his in December 2012, discovering a few tracks every few years following, until I saw him live in March this year. You can see what happened after that in this graph:

Interestingly enough, I went to that show in March 2019 to see Gilligan Moss, who made my top 10 artists last year and were my most popular newly-discovered artist of 2018. If I hadn’t discovered them last year, I probably wouldn’t have gone to that show at all, and this year would have been completely different.

Spotify claims that I spent 8 hours listening to Tourist this year. My own data? Rough calculations estimate that I’ve spent 15 hours and 15 minutes listening to Tourist. To put that in perspective, I spent at least 238 hours listening to music this year. At least 6% of my total listening time was spent on this one artist. Nice.¹

According to Spotify, my favorite Tourist song was “Too Late - Continuous Mix”. If I consolidate the two similar tracks in my data (data consistency is hard), that’s also true overall—that track has 24 listens so far (which would actually make it #4 on my top tracks of the year).

To move beyond Tourist, Spotify also told me that I discovered 1503 new artists, and that Plastic Plates were my favorite of those. Meanwhile in my data, I see that I discovered 2857 artists this year (probably at least 100 of those are random Youtube videos that got mis-filed), with my top 5 discoveries being:

Benoit & Sergio with 86 listens
Kölsch with 55 listens
warner case with 39 listens
Parra for Cuva with 38 listens
Lindstrøm with 28 listens

According to my data, I only listened to Plastic Plates 3 times this year after discovering them on January 2, 2019.

You can see more details about my new discoveries, along with a sparkline of my listening patterns for those artists throughout the year, in this table:

I spent 35,496 minutes listening to music this year, according to Spotify. Spotify’s data is much better than mine in this respect (100% coverage of metadata!) because my data tells me I only spent 14,296 minutes listening to music. In reality, it’s probably closer to the sum of those numbers.

What else happened in 2019 that Spotify doesn’t know about?

My top 10 albums of the year:

This is where you can see data struggles once again. Those two Tourist albums are essentially the same album, just differently-named in Spotify vs iTunes, so that is actually my most-listened-to album of the year. The Hood Internet metadata was incomplete when they shared their 1979-1983 mashup tracks on SoundCloud, so the free downloads that I added to my iTunes library show up without an album. Which is actually technically correct.

In addition to those top 10 albums, here is an area graph showing the total listens of my top 10 artists over 2019:

That looks somewhat exciting until you see those numbers stacked against all the other artists I listened to this year:

I added 344 new tracks (so far) to my iTunes library, and listened to 5,902 different tracks a total of 9,823 times (so far). I went to 49 concerts (with my 50th of the year lined up for tonight!) seeing a total of 136 artists (so far). Numbers!

My most frequented-venues of the year were 1015 Folsom and Audio, followed closely by the Fox Theater, Great American Music Hall, and The Fillmore. The artist I saw most frequently was Teh Raptor (DJ sets), soon to be tied by Tourist when I see him for the third time (in general and in 2019) in a couple weeks.

Spotify was also able to tell me some things that I can’t yet identify, namely that I listened to artists from 73 different countries. I’m hopeful that next year I’ll have additional metadata from the MusicBrainz database set up and correlating with my Splunk indexes.

The 2010s: Best of the Decade

Because 2019 is the last year of the decade, Spotify also added some stats for the entire decade to their #wrapped feature.

My top 5 artists of the 2010s according to Spotify are:

Daughter
Hey Rosetta!
CHVRCHES
Cold War Kids
The Format

According to my data, these artists are my top 5 of the 2010s:

Hey Rosetta! with 1162 listens
Alkaline Trio with 803 listens
Cold War Kids with 743 listens
Manchester Orchestra with 721 listens
CHVRCHES with 674 listens

Motion City Soundtrack just barely missed out on the top 5, with 673 total listens. The Format, meanwhile are in 10th with 493 total listens. Daughter didn’t make my top 10, but are instead 24th with 314 total listens for the decade.

My top 5 songs of the decade according to Spotify are:

According to my own data:

Hey Rosetta! - Trish’s Song | 114 listens
Hey Rosetta! - Kintsukuroi | 102 listens
Hey Rosetta! - The Simplest Thing | 95 listens
Carly Rae Jepsen - Run Away With Me | 93 listens
CHVRCHES - High Enough to Carry You Over | 90 listens

It’s probably then no surprise to learn that my top album of the decade is Hey Rosetta!’s Second Sight, which features the first 2 songs of my top 5 of the decade, and came out in 2014. My other top albums of the decade:

According to Spotify I’ve been using their service since 2011, but I think it’s actually more like 2013—and this is borne out in their data. They list my top tracks and artists for the decade only starting in 2013. Let’s compare!

Year	Spotify	Total Listens	My Data	Total Listens
2010	-	-	Hello Saferide - The Quiz	56
2011	-	-	Jamie xx - Rolling in the Deep remix feat. Childish Gambino	49
2012	-	-	Smoking Popes - Can’t Find It	33
2013	Daughter - Winter	1	Cold War Kids - Bulldozer	16
2014	Daughter - Youth	4	Taylor Swift - Blank Space	31
2015	Hey Rosetta! - The Simplest Thing	53	Hey Rosetta! - Kintsukuroi	55
2016	Carly Rae Jepsen - Run Away With Me	78	Carly Rae Jepsen - Run Away With Me	78
2017	Gibbz - 24/7	18	Hey Rosetta! - Trish’s Song Sjowgren - Now & Then	26 24
2018	Jude Woodhead - Beautiful Rain	25	Young Fathers - Border Girl	35
2019	Manatee Commune - W/O	10	Hey Rosetta! - Trish’s Song The Chemical Brothers - Got To Keep On	28 25

I added extra lines for the years when “Trish’s Song” took the top spot because that song is a lullaby and I listen to it accordingly—so perhaps I should consider the second place song as the “true” top song for that year.

Fun fact, my fifth-most-listened-to track of 2014 is The Riff-Off from Pitch Perfect. I watched it so many times on YouTube y’all now have some idea how obsessed I was (am).

In 2016, Run Away With Me by Carly Rae Jepsen took the top track slot by 1 listen. Ariana Grande’s Into You trailed with 77. Those 2 tracks were on a playlist of only 4 tracks that I listened to a LOT that year. The other 2 tracks from the playlist were my 4th- and 5th-most-listened tracks of the year: Ingrid Michaelson’s Hell No with 52 listens and Adele’s Send My Love (To Your New Lover), also with 52 listens.

My top artists for each year of the past decade are as follows, comparing Spotify’s data with my data. I gotta say, I wasn’t expecting to see Taylor Swift take the top spot for 2014.

Year	Spotify	Total Listens	My Data	Total Listens
2010	-	-	Alkaline Trio	352
2011	-	-	Tegan and Sara	236
2012	-	-	Smoking Popes	241
2013	The Format	9	Cold War Kids	57
2014	The Format	69	Taylor Swift	108
2015	Hey Rosetta!	-	Hey Rosetta!	591
2016	Jason Derulo	179	Hey Rosetta!	332
2017	Cold War Kids	156	The xx	166
2018	Poolside	-	Poolside	162
2019	Tourist	-	Tourist	251

My music listening habits (and possibly also my data fidelity) dropped dramatically in the early/mid-2010s, which is why those numbers are so different compared with other years.

It’s fun to see my overall trend in top artists for the decade. It’s almost like the 2013/2014 dropoff in music listening also coincided with a pivot in terms of what artists I was listening to.

I was also in college until 2012, but Manchester Orchestra and Alkaline Trio and Someone Still Loves You Boris Yeltsin almost totally drop out of the listening patterns after 2013, taken over by Hey Rosetta!, CHVRCHES, The Format, and The xx resurging in 2017.

In total I listened to 27,068 unique tracks 100,240 times since 2010, spending 223,350 minutes (at least) listening to music. I purchased a total of 772 songs from the iTunes store in the last decade, with nearly half of those purchases happening this year.

Considering those total minutes spent listening, Spotify also shared minutes spent listening data all the way back to 2015! I’ve already talked about why my data is so different from Spotify’s (~ incomplete metadata ~) but here’s how the numbers compare:

Year	Spotify	My Data
2015	11,834	20,462
2016	28,659	19,959
2017	26,137	13,919
2018	35,655	16,737
2019	35,496	14,296

It’s fun to review the artists that I’ve discovered in the past decade, with Hey Rosetta!, CHVRCHES, Mumford & Sons, Two Door Cinema Club, and Daughter taking the top 5 spots.

I’ve attended 163 concerts so far in the last decade, seeing a total of 404 artists The distribution of those concerts and artists over time is interesting to look at as well: spikes while I was in college, but not really taking off until I moved to San Francisco and joined a concert community group in the area.

I saw several artists multiple times throughout the decade, some as supporting acts (Future Feats, who I saw as an opening act 3 times, despite not enjoying their sets) and others as a combination of supporting and main acts (such as Smoking Popes).

My most-visited venue of the decade was The Independent, which I’ve been to 14 times. I don’t think I made it to a single show there in 2019, but hopefully I’ll be back for a 15th visit soon.

This has been a lot of data. I shared a similar roundup last year around this time, My 2018 Year in Music: Data Analysis and Insights. It’s fascinating to look back at the entire decade and reflect on how my life has changed, how my music taste and listening habits have shifted (or not) over time, and see the influence of live music attendance in my listening patterns and popular artists. Whether I’m using Spotify, iTunes, the Music app on my phone, SoundCloud, YouTube, or seeing live music, I’m glad I have music in my life.

I calculated this by counting the listens for specific tracks in my Last.fm data, then looking up the lengths of those tracks in my iTunes data and multiplying the number of listens by the track lengths. Of course this means that I’m not even considering the tracks that aren’t in my library, since I’m missing that metadata. ↩︎

Unbiased data analysis with the data-to-everything platform: unpacking the Splunk rebrand in an era of ethical data concerns

Wed, 06 Nov 2019 16:20:37 +0000

Splunk software provides powerful data collection, analysis, and reporting functionality. The new slogan, “data is for doing”, alongside taglines like “the data-to-everything platform” and “turn data into answers” want to bring the company to the forefront of data powerhouses, where it rightly belongs (I’m biased, I work for Splunk).

There is nuance in those phrases that can’t be adequately expressed in marketing materials, but that are crucial for doing ethical and unbiased data analysis, helping you find ultimately better answers with your data and do even better things with it.

Start with the question

If you start attempting to analyze data without an understanding of a question you’re trying to answer, you’re going to have a bad time. This is something I really appreciate about moving away from the slogan “listen to your data” (even though I love a good music pun). Listening to your data implies that you should start with the data, when in fact you should start with what you want to know and why you want to know it. You start with a question.

Data analysis starts with a question, and because I’m me, I want to answer a fairly complex question: what kind of music do I like to listen to? This overall question, also called an objective function in data science, can direct my data analysis. But first, I want to evaluate my question. If I’m going to turn my data into doing, I want to consider the ethics and the bias of my question.

Consider what you want to know, and why you want to know it so that you can consider the ethics of the question.

Is this question ethical to ask?
Is it ethical to use data to answer it?
Could you ask a different question that would be more ethical and still help you find useful, actionable answers?
Does my question contain inherent bias?
How might the biases in my question affect the results of my data analysis?

Questions like “How can we identify fans of this artist so that we can charge them more money for tickets?” or “What’s the highest fee that we can add to tickets where people will still buy the tickets?” could be good for business, or help increase profits, but they’re unethical. You’d be using data to take actions that are unfair, unequal, and unethical. Just because Splunk software can help you bring data to everything doesn’t mean that you should.

Break down the question into answerable pieces

If my question is something that I’ve considered ethical to use data to help answer, then it’s time to consider how I’ll perform my data analysis. I want to be sure I consider the following about my question, before I try to answer it:

Is this question small enough to answer with data?
What data do I need to help me answer this question?
How much data do I need to help me answer this question?

I can turn data into answers, but I have to be careful about the answers that I look for. If I don’t consider the small questions that make up the big question, I might end up with biased answers. (For more on this, see my .conf17 talk with Celeste Tretto).

So if I consider “What kind of music do I like to listen to?”, I might recognize right away that the question is too broad. There are many things that could change the answer to that question. I’ll want to consider how my subjective preferences (what I like listening to) might change depending on what I’m doing at the time: commuting, working out, writing technical documentation, or hanging out on the couch. I need to break the question down further.

A list of questions that might help me answer my overall question could be:

What music do I listen to while I’m working? When am I usually working?
What music do I listen to while I’m commuting? When am I usually commuting?
What music do I listen to when I’m relaxing? When am I usually relaxing?
What are some characteristics of the music that I listen to?
What music do I listen to more frequently than other music?
What music have I purchased or added to a library?
What information about my music taste isn’t captured in data?
Do I like all the music that I listen to?

As I’m breaking down the larger question of “What kind of music do I like to listen to?”, the most important question I can ask is “What kind of music do I think I like to listen to?”. This question matters because data analysis isn’t as simple as turning data into answers. That can make for catchy marketing, but the nuance here lies in using the data you have to reduce uncertainty about what you think the answer might be. The book How to Measure Anything by Douglas Hubbard covers this concept of data analysis as uncertainty reduction in great detail, but essentially the crux is that for a sufficiently valuable and complex question, there is no single objective answer (or else we would’ve found it already!).

So I must consider, right at the start, what I think the answer (or answers) to my overall question might be. Since I want to know what kind of music I like, I therefore want to ask myself what kind of music I think I might like. Because “liking” and “kind of music” are subjective characteristics, there can be no single true answer that is objective truth. Very few, if any, complex questions have objectively true answers, especially those that can be found in data.

So I can’t turn data into answers for my overall question, “What kind of music do I like?” but I can turn it into answers for more simple questions that are rooted in fact. The questions I listed earlier are much easier to answer with data, with relative certainty, because I broke up the complex, somewhat subjective question into many objective questions.

Consider the data you have

After you have your questions, look for the answers! Consider the data that you have, and whether or not it is sufficient and appropriate to answer the questions.

The flexibility of Splunk software means that you don’t have to consider the questions you’ll ask of the data before you ingest it. Structured or unstructured, you can ask questions of your data, but you might have to work harder to fully understand the context of the data to accurately interpret it.

Before you analyze and interpret the data, you’ll want to gather context about the data, like:

Is the dataset complete? If not, what data is missing?
Is the data correct? If not, in what ways could it be biased or inaccurate?
Is the data similar to other datasets you’re using? If not, how is it different?

This additional metadata (data about your datasets) can provide crucial context necessary to accurately analyze and interpret data in an unbiased way. For example, if I know there is data missing in my analysis, I need to consider how to account for that missing data. I can add additional (relevant and useful) data, or I can acknowledge how the missing data might or might not affect the answers I get.

After gathering context about your datasets, you’ll also want to consider if the data is appropriate to answer the question(s) that you want to answer.

In my case, I’ll want to assess the following aspects of the datasets:

Is using the audio features API data from Spotify the best way to identify characteristics in music I listen to?
Could another dataset be better?
Should I make my own dataset?
Does the data available to me align with what matters for my data analysis?

You can see a small way that the journalist Matt Daniels of The Pudding considered the data relevant to answer the question “How popular is male falsetto?” for the Vox YouTube series Earworm starting at 1:45 in this clip. For about 90 seconds, Matt and the host of the show, Estelle Caswell, discuss the process of selecting the right data to answer their question, including discussing the size of the dataset (eventually choosing a smaller, but more relevant, dataset) to answer their question.

Is more data always better?

Data is valuable when it’s in context and applied with consideration for the problem that I’m trying to solve. Collecting data about my schedule may seem overly-intrusive or irrelevant, but if it’s applied to a broader question of “what kind of music do I like to listen to?” it can add valuable insights and possibly shift the possible overall answer, because I’ve applied that additional data with consideration for the question that I’m trying to answer.

Splunk published a white paper to accompany the rebranding, and it contains some excellent points. One of them that I want to explore further is the question:

“how complete, how smart, are these decisions if you’re ignoring vast swaths of your data?”

On the one hand, having more data available can be valuable. I am able to get a more valuable answer to “what kind of music do I like” because I’m able to consider additional, seemingly irrelevant data about how I spend my time while I’m listening to music. However, there are many times when you want to ignore vast swaths of your data.

The most important aspect to consider when adding data to your analysis is not quantity, but quality. Rather than focusing on how much data you might be ignoring, I’d suggest instead focusing on which data you might be ignoring, for which questions, and affecting which answers. You might have a lot of ignored data, but put your focus on the small amount of data that can make a big difference in the answers you find in the data.

As the academics in “I got more data, my model is more refined, but my estimator is getting worse! Am I just dumb?” make clear with their crucial finding:

“More data lead to better conclusions only when we know how to take advantage of their information. In other words, size does matter, but only if it is used appropriately.”

The most important aspect of adding data to an analysis is exactly as the academics point out: it’s only more helpful if you know what to do with it. If you aren’t sure how to use additional data you have access to, it can distract you from what you’re trying to answer, or even make it harder to find useful answers because of the scale of the data you’re attempting to analyze.

Douglas Hubbard in the book How to Measure Anything makes the case that doing data analysis is not about gathering the most data possible to produce the best answer possible. Instead, it’s about measuring to reduce uncertainty in the possible answers and measuring only what you need to know to make a better decision (based on the results of your data analysis). As a result, such a focused analysis often doesn’t require large amounts of data — rough calculations and small samples of data are often enough. More data might lead to greater precision in your answer, but it’s a tradeoff between time, effort, cost, and precision. (I also blogged about the high-level concepts in the book).

If I want to answer my question “What kind of music do I like to listen to?” I don’t need the listening data of every user on the Last.fm service, nor do I need metadata for songs I’ve never heard to help me identify song characteristics I might like. Because I want to answer a specific question, it’s important that I identify the specific data that I need to answer it—restricted by affected user, existence in another dataset, time range, type, or whatever else.

If you want more evidence, the notion that more data is always better is also neatly upended by the Nielsen-Norman Group in Why You Only Need to Test with 5 Users and the follow-up How Many Test Users in a Usability Study?.

Keep context alongside the data

Indeed, the white paper talks about bringing people to a world where they can take action without worrying about where their data is, or where it comes from. But it’s important to still consider where the data comes from, even if you aren’t having to worry about it because you use Splunk software. It’s relevant to data analysis to keep context about the data alongside the data.

For example, it’s important for me to keep track of the fact that the song characteristics I might use to identify the type of music I like come from a dataset crafted by Spotify, or that my listening behavior is tracked by the service Last.fm. Last.fm can only track certain types of listening behavior on certain devices, and Spotify has their own biases in creating a set of audio characteristics.

If I lose track of this seemingly-mundane context when analyzing my data, I can potentially incorrectly interpret my data and/or draw inaccurate conclusions about what kind of music I like to listen to, based purely on the limitations of the data available to me. If I don’t know where my data is coming from, or what it represents, then it’s easy to find biased answers to questions, even though I’m using data to answer them.

If you have more data than you need, this also makes keeping context close to your data more difficult. The more data, the more room for error when trying to track contextual meaning. Splunk software includes metadata fields for data that can help you keep some context with the data, such as where it came from, but other types of context you’d need to track yourself.

More data can not only complicate your analysis, but it can also create security and privacy concerns if you keep a lot of data around and for longer than you need it. If I want to know what kind of music I like to listen to, I might be comfortable doing data analysis to answer that question, identifying the characteristics of music that I like, and then removing all of the raw data that led me to that conclusion out of privacy or security concerns. Or I could drop the metadata for all songs that I’ve ever listened to, and keep only the metadata for some songs. I’d want to consider, again, how much data I really need to keep around.

Turn data into answers—mostly

So I’ve broken down my overall question into smaller, more answerable questions, I’ve considered the data I have, and I’ve kept the context alongside the data I have. Now I can finally turn it into answers, just like I was promised!

It turns out I can take a corpus of my personal listening data and combine it with a dataset of my personal music libraries to weight the songs in the listening dataset. I can also assess the frequency of listens to further weight the songs in my analysis and formulate a ranking of songs in order of how much I like them. I’d probably also want to split that ranking by what I was doing while I was listening to the music, to eliminate outliers from the dataset that might bias the results. All the small questions that feed into the overall question are coming to life.

After I have that ranking, I could use additional metadata from another source, such as the Spotify audio features API, to identify the characteristics of the top-ranked songs, and ostensibly then be able to answer my overall question: what kind of music do I like to listen to?

By following all these steps, I turned my data into answers! And now I can turn my data into doing, by taking action on those characteristics. I can of course seek out new music based on those characteristics, but I can also book the ideal DJs for my birthday party, create or join a community of music lovers with similar taste in music, or even delete any music from my library that doesn’t match those characteristics. Maybe the only action I would take is self-reflection, and see if what the data has “told” me is in line with what I think is true about myself.

It is possible to turn data into answers, and turn data into doing, with caution and attention to all the ways that bias can be introduced into the data analysis process. But there’s still one more way that data analysis could result in biased outcomes: communicating results.

Carefully communicate data findings

After I find the answers in my data, I need to carefully communicate them to avoid bias. If I want to tell all my friends that I figured out what kind of music I like to listen to, I want to make sure that I’m telling them that carefully so that they can take the appropriate and ethical action in response to what I tell them.

I’ll want to present the answers in context. I need to describe the findings with the relevant qualifiers: I like music with these specific characteristics, and when I say I like this music I mean this is the kind of music that I listen to while doing things I enjoy, like working out, writing, or sitting on my couch.

I also need to make clear what kind of action might be appropriate or ethical to take in reaction to this information. Maybe I want to find more music that has these characteristics, or I’d like to expand my taste, or I want to see some live shows and DJ sets that would feature music that has these characteristics. Actions that support those ends would be appropriate, but can also risk being unethical. What if someone learns of these characteristics, and chooses to then charge me more money than other people (whose taste in music is unknown) to see specific DJ sets or concerts featuring music with those characteristics?

Data, per the white paper, “must be brought not only to every action and decision, but to every department.” Because of that, it’s important to consider how that happens. Share relevant parts of the process that led to the answers you found from the data. Communicate the results in a way that can be easily understood by your audience. This Medium post by Cecelia Shao, a product manager at Comet.ml, covers important points about how to communicate the results of data analysis.

Use data for good

I wanted to talk through the data analysis process in the context of the rebranded slogans and marketing content so that I could unpack additional nuance that marketing content can’t convey. I know how easy it is to introduce bias into data analysis, and how easily data analysis can be applied to unethical questions, or used to take unethical actions.

As the white paper aptly points out, the value of data is not merely in having it, but in how you use it to create positive outcomes. You need to be sure you’re using data safely and intelligently, because with great access to data comes great responsibility.

Go forth and use the data-to-everything platform to turn data into doing…the right thing.

Disclosure: I work for Splunk. Thanks to my colleagues Chris Gales, Erica Chen, and Richard Brewer-Hay for the feedback on drafts of this post. While colleagues reviewed this post and provided feedback, the content is my own and represents my own views rather than those of Splunk the company.

Streaming, the cloud, and music interactions: are libraries a thing of the past?

Tue, 29 Oct 2019 07:18:59 +0000

Several years ago I wrote about fragmented music libraries and music discovery. In light of the overwhelming popularity of Spotify and the dominance of streaming music (Spotify, Apple Music, Amazon Music, Tidal, and others), I’m curious if music libraries even exist anymore. Or, if they exist today, will they continue to exist?

My guess is that the only people still maintaining music libraries are DJs, fervent music fans (like myself), or people that aren’t using streaming music at all (due to age, lack of interest, or lack of availability due to markets or internet speeds).

I was chatting with a friend of mine that has a collection of vinyl records, but she only ever listens to vinyl if she’s relaxing on the weekend. Oftentimes she’s just asking Alexa to play some music, without much attention to where that music is coming from. With Amazon Music bundled into Amazon Prime for many members, people can be totally unaware that they’re using a streaming service at all. I’d hazard that this interaction pattern is true for most people, especially those that never enjoyed maintaining a music library but instead collected CDs and records because that was the only way to be able to listen to music at all.

Even my own habits are changing, perhaps equally due to time constraints as due to current music technology services. I used to carefully curate playlists for sharing with others, listening in the car, mix CDs, and for radio shows. These days I make playlists for many of those same purposes on Spotify, but the songs in my “actual” music library (iTunes) aren’t categorized into playlists at all anymore, and I give the playlists I make on my iPhone random names like “Aaa yay” to make the playlists easier to find, rather than to describe the contents.

I’m limited by storage size in terms of what I can add to my iPhone, just like I was with my iPod, but that shapes my experience of the music. Since I’m limited to a smaller catalogue, I’m able to sit with the music more and create more distinct memories. There are still songs that remind me of being in Berlin in 2011, limited to the songs that I added to my iPod before I left the United States because the internet I had access to in Germany was too slow to download new music and add it to my iPod.

Nowadays, I am less motivated to carefully manage my iTunes library because it’s only on one device, whereas I can access my Spotify library across multiple devices. That’s the one I find myself carefully creating folders of playlists for, organizing and sorting tracks and playlists. A primary reason for the success of Spotify for my listening habits is the social and collaborative nature of it. It’s easy to share tracks with others, make a playlist for a DJ set that I went to to share with others, contribute to a weekly collaborative playlist with a community of fellow music-lovers, or to follow playlists created by artists and DJs I love. My local library can give me a lot, but it can’t give me that community interaction.

Indeed, in 2015 that’s something I identified as lacking. I felt that it was harder to feel part of a music culture, writing:

“It’s harder than it used to be to feel connected with music. It’s not a stream or a subculture one is tapped into anymore, because it’s so distributed on the web. There’s so much music, and it lives in so many different services, that the music culture has imploded a bit.”

I feel completely differently these days, thanks to a vibrant live music community in San Francisco. I loathe Facebook, but the groups that I’m a part of on that site enable me to feel connected to a greater music scene and community that supplement my connection to music and music discovery. Ironically, Facebook groups have also helped my music culture experience become more local. The music blogs that I used to be able to tap into are now largely defunct, or have multiple functions (the burning ear also running vinyl me please, or All Things Go also providing news and an annual festival in DC). Instead yet another way I discover new music is by paying attention to the artists and DJs that people in these Facebook groups are talking about and posting tracks and albums from.

Despite the challenges of a local music library, I keep buying digital music partially because I made a promise to myself when I was younger that I’d do so when I could afford to, partially to support musicians and producers, and partially because I distrust that streaming services will stick around with all the music I might want to listen to. I’d rather “own” it, at least as best as I can when it’s a digital file that risks deletion and decomposition over time.

Music discovery in the past was equal parts discovery and collection, with a hefty dose of listening after I collected new music.

I’d do the following when discovering new music:

Write down song lyrics while listening to the radio or while working my retail job, then later looking up the tracks to check out albums from the library to rip to my family computer.
Follow music blogs like The Burning Ear, All Things Go, Earmilk, Stereogum, Line of Best Fit, then downloading what I liked best from their site from MediaFire or MegaUpload to save to my own library.
Trawl through illicit LiveJournal communities or invite-only torrent sites to download discographies for artists I already liked, or might like.

Over time, those music blogs shifted to using SoundCloud, the online communities and torrent sites shuttered, and I started listening to more music on streaming sites instead. The loop stopped going from discovery to collection and instead to discovery, like, and discovery again.

Find a new track, listen, click the heart or the plus sign, and move on. Rarely do you remember to go back and listen to your fully-compiled list of saved tracks (or even if you do, trying to listen to the whole thing on shuffle will be limited by the web app, thanks SoundCloud).

This type of cycle is faster than the old cycle, and more focused on engagement with the service (rather than the music) and less on collecting and more on consuming. In some ways, downloading music was like this too. When I accidentally deleted my entire music library in 2012, the tatters of my library that I was able to recover from my iPod was a scant representation of my full collection, but included in that library was discographies that I would likely never listen to. Now that it’s been years, there have been a few occasions where I go back and discover that an artist I listen to now is in that graveyard of deleted songs, but even knowing that, I’m not sure I would’ve gotten to it any sooner. I was always collecting more than I was listening to.

Streaming music lets me collect in the same way, but without the personal risk. It just makes me dependent on a third-party entity that permits me to access the tracks that they store for me. I end up with lists of liked tracks across multiple different services, none of which I fully control. These days my music discovery is now largely driven by 3 services: Spotify, Shazam, and Soundcloud. Spotify pushes algorithmic recommendations to me, Shazam enables me to discover what track the DJ is currently playing when I’m out at a DJ set, and Soundcloud lets me listen to recorded DJ sets as well as having excellent autoplay recommendations. In all of them I have lists of tracks that I may never revisit after saving them. Some of them I’ll never be able to revisit, because they’ve been deleted or the service has lost the rights to the track.

In 2015 I lamented the fragmentation of music discovery, but looking back, my music discovery was always shared across services, devices, and methods—the central iTunes library was what tied the radio songs, the library CDs, the discography downloads, and the music blog tracks together. The real issue is that the primary music discovery modes of today are service-dependent, and each of those services provides their own constructs of a music library. I mentioned in 2015 that:

“my library is all over the place. iTunes is still the main home of my music—I can afford to buy new music when I want —but I frequent Spotify and SoundCloud to check out new music. I sync my iTunes library to Google Play Music too, so I can listen to it at work.”

While this is still largely true, I largely consume Spotify when I’m at work, listen to SoundCloud sets or tracks from iTunes when I’m on-the-go with my phone, and listen to Spotify or iTunes when I’m on my personal laptop. That’s essentially 2.5 places that I keep a music library, and while I maintain a purchase pipeline of tracks from Spotify and SoundCloud into my iTunes library, it’s a fraction of my discoveries that make it into my collection for the long term. The days of a true central collection of my library are long since past.

It seems a feat, with all these digital cloud music services streaming music into our ears, to have a local music library. Indeed, what’s the point of holding onto your local files when it becomes so difficult to access it? iTunes is becoming the Apple Music app, with the Apple Music streaming service front and center. Spotify is, well, Spotify. And SoundCloud continues to flounder yet provides an essential service of underground music and DJ sets. Google Play Music exists, but only has a web-based player (no client) to make it easier to access and listen to your local library after you’ve mirrored it to the cloud. Streaming is convenient. But streaming music lets others own your content for you, granting you subscription access to it at best, ruining the quality of your music listening experience at worst. A recent essay by Dave Holmes in Esquire talks about “The Deleted Years”, or the years that we stored music on iPods, but since Spotify and other streaming services, have largely moved on from. As he puts it,

“From 2003 to 2012, music was disposable and nothing survived.”

Perhaps it’s more true that from 2012 onward, music is omnipresent and yet more disposable. It can disappear into the void of a streaming service, and we’ll never even know we saved it. At least an abandoned iPod gives us a tangible record of our past habits. As Vicki Boykis wrote about SoundCloud in 2017,

“I’m worried that, for internet music culture, what’s coming is the loss of a place that offered innumerable avenues for creativity, for enjoyment, for discovery of music that couldn’t and wouldn’t be created anywhere else. And, like everyone who has ever invested enough emotion in an online space long enough to make it their own, I’m wondering what’s next.”

I’ll be here, discovering, collecting, liking, and listening for what’s next.

Music streaming and sovereignty

Wed, 29 May 2019 19:25:15 +0000

As the music industry moves away from downloads and toward building streaming platforms, international sovereignty becomes more of a barrier to people listening to music and discussing it with others, because they don’t have access to the same music on the same platforms. As Sean Michaels points out in The Morning News several years ago:

one of the undocumented glitches in the current internet is all its asymmetrical licensing rules. I can’t use Spotify in Canada (yet). Whenever I’m able to, there’s no guarantee that Spotify Canada’s music library will match Spotify America’s. Just as Netflix Canada is different than Netflix US, and YouTube won’t let me see Jon Stewart. As we move away from downloads and toward streaming, international sovereignty is going to become more and more of a barrier to common discussions of music.

Location has always been a challenge to music access, but it’s important to keep in mind that the internet and music streaming has not been an equitable boon to music access—it is still controlled.

Defining my career values

Fri, 24 May 2019 18:02:10 +0000

If you’re thinking about changing careers, or want guidance in determining whether your career is right for you, I hope this post can help you! It’s all about how I defined my career values and reframed how I thought about my career and my future.

Why I needed to define my career values

A couple years ago I was comfortable in my position at work. After four years in my career, surrounded by talk of the importance of having a growth mindset, I thought maybe I was too comfortable.

As a technical writer, I was contributing to product management conversations, and thinking intensely about customer needs, and realized I wanted to be even more involved in what we were choosing to build.

I took a training course, and found a job on the product management team in my company that appealed to my interests. 11 months later, I went back to documentation, after realizing that that pathway better suited my career values.

Throughout those 11 months and in the time since, I’ve worked to determine what I really want to get out of my career, and make sure that what I am doing fits those values.

Ask myself some questions

I started by asking myself some questions, common ones that people recommend when you’re thinking about making a job change. I found that I was better able to answer these questions after I’d already made a job change, likely because I didn’t have that much work experience before making the career change.

I asked myself the following questions:

What makes me excited to go into work?
What makes me dread going into work?
What helps me feel validated or appreciated at work?
- Working with others? Contributing in meetings? Reporting project status on a regular basis?
What is my working style when working with others?
- Do I prefer collaborative, consultative, or independent work?
What do I like producing when I’m at work?
- Ideas, or tangible things? Concrete concepts or future-oriented concepts?
Do I prefer hands-on management, consultative management, or completely hands-off management?

After changing roles, I realized that many parts of my technical writing position, and the way that my team and my duties were structured, were very well suited to my working styles. However, since I hadn’t had much experience with other types of work, I hadn’t identified them as vital to my work. Switching positions forced me to reexamine what parts of a role were vital to my happiness at work, and in what way.

Find strategies that have worked for others

I found several strategies that worked for others by listening to some You 2.0 episodes from the Hidden Brain podcast.

I realized there were options to transform a job I was already in by finding more enjoyable aspects within it by listening to the You 2.0 Dream Jobs and You 2.0 How to Build a Better Job podcast episodes of Hidden Brain.

The dream jobs episode helped me consider whether I was looking for too much meaning and validation within my job, and if I needed to separate those pursuits more. The how to build a better job helped me consider what I could shift within my day-to-day job in terms of focus or duties so that I could enjoy it as well.

I also looked beyond work-specific strategies. The episode You 2.0: How Silicon Valley Can Help You Get Unstuck taught me perhaps my favorite tidbit, which is to apply iterative methodologies to your life.

Yes it’s kitschy, but one example he mentioned resonated deeply with me: the notion of creating multiple five year plans. Whenever I’d previously considered how my future might look, it was easy to get stressed about the fact that I have one future available to me and ~ people ~ expect me to have a plan for it. But this philosophy helped me realize that I can have multiple plans for it, and test them out.

I put together several five year plans to speculate about where I spend my time and how my life might look if I stayed pursuing product management roles, or how it might look if I was doing tech writing for those years, as well as what it might look like if I took a different role entirely, moved cities, or even moved countries.

This helped me consider what types of futures excited me, and position my work priorities alongside my overall life priorities. They aren’t separate, and I wanted to be sure that I didn’t consider them separately.

I also realized that this exercise wasn’t about making these plans and then choosing one of them, but rather choosing the elements of each of the plans that made sense to me and got me excited about the future. I plan to revisit this exercise and continue to evaluate the spectrum of futures available to me.

The episode You 2.0: Decide Already! interviewed Dan Gilbert, the author of the excellent book Stumbling on Happiness. Both the episode and the book helped me consider the ways that being anxious about the future and planning for it and attempting to reduce uncertainty about it wasn’t necessarily making me feel better about it—and might actually be making me feel worse.

Despite creating some five year plans, allowing room and flexibility in those plans, and welcoming uncertainty in my work and life is also crucial.

Take a values-centered approach

In my personal life I was working on developing and defining my personal values, using a card-sorting exercise similar to this one from the Urban Indian Health Institute. Defining my personal values, and understanding them as a way to assess whether or not my goals and day-to-day tasks were fulfilling or not, turned out to be vital.

I attempted to apply a similar framework to my work goals and fulfillment as well, and identify one or more overarching themes that I could associate with my career.

Putting all the strategies together to define career values

After assessing the structure of work that I thrive and find validation in, I was better able to understand what I found fulfilling about a career, and what I could look for in future roles to find fulfillment and the right kind of comfort:

A work environment with clear expectations and measurable, tangible results, was vital.
A team that I could collaborate with and draw support from, while also working semi-independently, was also important to me.

After creating multiple five year plans, I was able to realize that a career path more similar to the one I had as a technical writer was more valuable to me than one that was closer to product management, where I’d be busier and spending more time and stress on work than on my personal life. In addition, by engaging with the technical writer community, I realized that the futures available to me with a technical writing career were more broad, varied, and flexible than I’d previously realized.

I didn’t need the power and recognition within a company that a product management position might offer me, because that power and recognition would also come with added responsibilities, time commitments, and stressful challenges.

I attempted to reverse-engineer my career values based on these experiences and my personal values exercise. I ultimately centered on a core career value of “Information Conveyance”.

What this means to me is that if I spend my time at work learning and sharing information with others, I will likely feel fulfilled and be excited to go to work. Defining this as a career value allowed me to move past specific roles and titles, because multiple career paths can help me support this value.

Right now I love technical writing, but other functions like communication strategy, developer advocacy, community management, instructional designer, and others align with this value and are available to me as other potential career paths.

Detailed data types you can use for documentation prioritization

Tue, 21 May 2019 17:16:22 +0000

Data analysis is a valuable way to learn more about what documentation tasks to prioritize above others. My post (and talk), Just Add Data, presented at Write the Docs Portland in 2019, talk about this broadly. In this post I want to cover in detail a number of different data types that can lead to valuable insights for prioritization.

This list of data types is long, but I promise each one contains value for a technical writer. These types of data might come from your own collection, a user research organization, the business development department, marketing organization, or product management organization:

User research reports
Support cases
Forum threads and questions
Product usage metrics
Search strings
Tags on bugs or issues
Education/training course content and questions
Customer satisfaction survey

User research reports

User research reports can contain a lot of valuable data that you can use for documentation.

Types of customers being interviewed
Customer use cases and problems
Types of studies being performed

This can give you insight into both what the company finds valuable to study (so some insight into internal priorities) but also direct customer feedback about things that are confusing or the ways that they use the product. The types of customers that are interviewed can provide valuable audience or persona-targeting information, allowing you to better calibrate the information in your documentation. See How to use data in user research when you have no web analytics on the Gov.UK site for more details about what you can do with user research data.

Support cases

Support cases can help you better understand customer problems. Specific metrics include:

Number of cases
Frequency of cases
Categories of questions
Customer environments and licenses

With these you can compile metrics about specific customer problems, the frequency of problems, and the types of customers and customer environments that are encountering specific problems, allowing you to better understand target customers, or customers that might be using your documentation more than others. Support cases are also rich data for common customer problems, providing a good way to gather new use cases and subjects for topics.

Forum threads and questions

These can be internal forums (like Splunk Answers for Splunk) or external ones, like Reddit or StackOverflow.

Common questions
Common categories
Frequently unanswered questions
Post titles

If you’re trying to understand what people are struggling with, or get a better sense of how people are using specific functionality, forum threads can help you understand. The types of questions that people ask and how they phrase them can also help make it clear what kinds of configuration combinations might make specific functions harder for customers. Based on the question types and frequencies that you see, you might be able to fine-tune existing documentation to make it more user-centric and easily findable, or supplement content with additional specific examples.

Product usage metrics

Some examples of product usage metrics are as follows:

Time in product
Intra-product clicks
Types of data ingested
Types of content created
Amount of content created

Even if you don’t have specific usage data introspecting the product, you can gather metrics about how people are interacting with the purchase and activation process, and extrapolate accordingly.

Number of downloads and installs
License activations and types
Daily and monthly active users

You can use this type of data to better understand how people are spending their time in your product, and what features or functionality they’re using. Even if a customer has purchased or installed the product, it’s even more valuable to find out if they’re actually using it, and if so, how.

If your product is only in beta, and you want more data to help you prioritize an overall documentation backlog, such as topics that are tied to a specific release, you can use some product usage data to understand where people are spending more of their time, and draw conclusions about what to prioritize based on that.

Maybe the under-utilized features could use more documentation, or more targeted documentation. Maybe the features themselves need work. Be careful not to draw overly-simplistic conclusions about the data that you see from product usage metrics. Keep context in mind at all times.

Search strings

You can gather search strings from HTTP referer data from web searches performed on external search sites such as Google or DuckDuckGo, or from internal search services. It’s pretty unlikely that you’ll be able to gather search strings from external sites given the widespread implementation of HTTPS, but internal search services can be vital and valuable data sources for this.

Look at specific search strings to find out what people are looking for, and what people are searching that’s landing them on specific documentation pages. Maybe they’re searching for something and landing on the wrong page, and you can update your topic titles to help.

JIRA or issue data

You can use metrics from your issue tracking services to better understand product quality, as well as customer confusion.

Number of issues/bugs
Categories/tags/components of issues/bugs
Frequency of different types of issues being created/closed

Issue tags or bug components can help you identify categories of the product where there are lots of problems or perhaps customer confusion. This is especially useful data if you’re an open source product and want to get a good understanding of where there are issues that might need more decision support or guidance in the documentation.

Training courses

If you have an education department, or produce training courses about your product, these are quite useful to gather data from. Some examples of data you might find useful:

Questions asked by customers
Questions asked by course developers
Use cases covered by content in courses
Enrollment in courses
Categories of courses offered

Also useful to correlate this with other data to help identify verticals of customers interested in different topics. Because education and training courses cover more hands-on material, it can be an excellent source of use case examples, as well as occasions where decision support and guidance is needed.

Customer surveys

Customer surveys especially cover surveys like satisfaction surveys and sentiment analysis surveys. By reviewing the qualitative statements and types of questions asked in the surveys, you can gain valuable insights and information like:

What do people think about the product?
What do people want more help with?
How do people think about the product?
How do people feel about the product?
What does the company want to know from customers?
What are the company priorities?

This can also help you think about how the documentation you write has a real effect on peoples’ interactions with the product, and can shift sentiment in one way or another.

Documentation feedback

Direct feedback on your documentation is a vital source of data if you can get it.

Qualitative comments about the documentation
Usefulness votes (yes/no)
Ratings

Even if you don’t have a direct feedback mechanism on your website, you can collect documentation feedback from internal and external customers by paying attention in conversations with people and even asking them directly if they have any documentation feedback. Qualitative comments and direct feedback can be vital for making improvements to specific areas.

Site metrics

If your documentation is on a website, you can use web access logs to gather important site metrics, such as the following:

Page views
Session data like time on page
Referer data
Link clicks
Button clicks
Bounce rate
Client IP

Site metrics like page views, session data, referer data, and link clicks can help you understand where people are coming to your docs from, how long they are staying on the page, how many readers there are, and where they’re going after they get to a topic. You can also use this data to understand better how people interact with your documentation. Are readers using a version switcher on your page? Are they expanding or collapsing information sections on the page to learn more? Maybe readers are using a table of contents to skip to specific parts of specific topics.

You can split this data by IP address to understand groups of topics that specific users are clustering around, to better understand how people use the documentation.

Text analysis metrics

Data about the actual text on your documentation site is also useful to help understand the complexity of the documentation on your site.

Flesch-Kincaid readability score
Inclusivity level
Length of sentences and headers
Style linter

You can assess the readability or usability of the documentation, or even the grade level score for the content to understand how consistent your documentation is. Identify the length of sentences and headers to see if they match best practices in the industry for writing on the web. You can even scan content against a style linter to identify inconsistencies of documentation topics against a style guide.

Download metrics

If you don’t have site metrics for your documentation site, because the documentation is published only via PDF or another medium, you can still use metrics from that.

Download numbers
Download dates and times
Download categories and types

You can use these metrics to gather interest about what people want to be reading offline, or how frequently people are accessing your documentation. You can also correlate this data with product usage data and release cycles to determine how frequently people access the documentation compared with release dates, and the number of people accessing the documentation compared with the number of people using a product or service.

Topic type metrics

If you use strict topic typing at your documentation organization, you can use topic type metrics as an additional metadata layer for documentation data analysis. Even if you don’t, you can manually categorize organize your documentation by type to gather this data.

What are the topic types?
How many topic types are there?
How many topics are there of each type?

Understanding topic types can help you understand how reader interaction patterns can vary for your documentation by type, or whether your developer documentation has predominantly different types of documentation compared with your user documentation, and better understand what types of documentation are written for which audiences.

Topic metadata

Metadata about documentation topics is also incredibly valuable as a correlation data source. You can correlate topic metadata like the following information:

What are the titles?
Average length of a topic?
Last updated and creation dates
Versions that different topics apply for

You can correlate it with site metrics, to see if longer topics are viewed less-frequently than shorter topics, or identify outliers in those data points. You can also manually analyze the topic titles to identify if there are patterns (good or bad) that exist.

Contribution data

If you have information about who is writing documentation, and when, you can use these types of data:

Last updated dates
Authors/contributors
Amount of information added or removed

Contribution data can tell you how frequently specific topics were updated to add new information, and by whom, and how much information was added or removed. You can identify frequency patterns, clusters over time, as well as consistent contributors.

It’s useful to split this data by other features, or correlate it with other metrics, especially site metrics. You can then identify things like:

Last updated dates by topic
Last updated dates by product
Last updated dates over time

to see if there are correlations between updates and page views. Perhaps more frequently updated content is viewed more often.

Social media referers
Link clicks from social media sites

If you publicize your documentation using social media, you can track the interest in the documentation from those sites. If you’re curious about social media referers leading people to your documentation, and see whether or not people are getting to your documentation in that way. Maybe your support team is responding to people on twitter with links to your documentation, and you want to better understand how frequently that happens and how frequently people click through those links to the documentation…

You can also identify whether or not, and how, people are sharing your documentation on social media by using data crawled or retrieved from those sites’ APIs, and looking for instances of links to your documentation. This can help you get a better sense of how people are using your documentation, how they’re talking about it, how they feel about it, and whether or not you have an organic community out there on the web sharing your documentation.

Beyond documentation data

I hope that this detail has given you a better understanding of different types of data, beyond documentation data, that are available to you as a technical writer to draw valuable conclusions from. By analyzing these types of data, you are prepared for prioritizing your documentation task list, but also better able to understand the customers of your product and documentation. Even if only some of these are available to you, I hope they are useful. Be sure to read Just Add Data: Using data to prioritize your documentationfor the full explanation of how to use data in this way.

Just Add Data: Using data to prioritize your documentation

Tue, 21 May 2019 17:15:40 +0000

This is a blog post adaptation of a talk I gave at Write the Docs Portland on May 21, 2019. The talk was livestreamed and recorded, and you can view the recording on YouTube: Just Add Data: Make it easier to prioritize your documentation - Sarah Moir

Prioritizing documentation is hard. How do you decide what to work on if there isn’t a deadline looming? How do you decide what not to work on when your list of work just keeps growing? How do you identify what new content you might want to add to your documentation?

By adding data to the process, it’s possible to prioritize your documentation tasks with confidence!

Prioritizing without data

Prioritizing a backlog without data can involve asking yourself some questions, like what will take the least amount of time? Or, what did someone most recently request? If I’m doing this, I might ask my product manager what to work on, or do whatever task seems easiest at the time. I might even focus on whichever task I can complete without talking to other people, because I’m tired.

Based on the answers to those questions, I’ll end up with a prioritized backlog, but lack confidence that what I’ve chosen to work on will actually bring the most value to customers and the documentation. Especially if I’m choosing not to do work, it can be a challenge to keep ignoring an item in the backlog because it doesn’t fit with what I think I need to be working on, especially without some sort of “proof” that it’s okay to ignore. To make this process easier, I add data.

Why prioritize with data?

Using data to prioritize a documentation backlog can help give you more confidence in your decisions and help you justify why you’re not working on something. It can challenge your assumptions about what you should be working on, or validate them. Adding data can help improve your overall understanding of how customers are using your product and the documentation, leading to benefits beyond the backlog.

Data types for prioritization

What kinds of data am I talking about? All kinds of data! If you skim the following list, you’ll notice that this data goes beyond quantitative sources. When I talk about data, I’m including all kinds of information: qualitative comments, usage metrics, metadata, website access logs, survey results, database records, all of these and more fit in with my definition of data. Here’s the full list

User research reports
Support cases
Forum threads and questions
Product usage metrics
Search strings
Tags on bugs or issues
Education/training course content and questions
Customer satisfaction survey
Documentation feedback
Site metrics
Text analysis metrics
Download/last accessed numbers
Topic type metrics
Topic metadata
Contribution data
Social media analytics

Some of these data types are more relevant to different types of organizations and documentation installations. For example, open source projects might have more useful issue tags, or organizations that use DITA will have easier access to topic type information.

This list of data types is to demonstrate the different types of information that can help you prioritize documentation, but I don’t want you to think that you need to do large-scale collections or implementations to get any valuable data worth incorporating into your prioritization process.

I’ll cover a couple of these data types in more detail here, but I talk about all of them in another post: Detailed data types you can use for documentation prioritization.

Product usage data

You can use usage data for products (also called telemetry) to find out where people are spending their time. What features or functionality are they using? Even if they’ve purchased or installed the product, are they actually using it?

Some examples of product usage data include:

Time in product
Intra-product clicks
Types of data ingested
Types of content created (e.g., dashboards, playlists)
Amount of content created (e.g., dashboards, playlists)

In addition to data about how people are interacting with the product, you can also gather product usage data without actual introspection into how people are using it. If you have information about how many people have downloaded a product or are logging in to a service:

Number of downloads and installs
License activations and types
Daily and monthly active users

I mostly talk about using data to help you prioritize the more ambiguous parts of a backlog that might not be tied to a release, but especially with the help of product usage data, you can better-prioritize release-focused documentation as well. If your product is in beta, and you want more data to help you prioritize your overall documentation backlog, you can use some product usage data to understand where people are spending more of their time, and draw conclusions about what to spend more time on or less time on, or what level of detail to include in the documentation, to achieve your overall documentation goals for the release.

Site metrics

Site metrics like page views, session data, HTTP referer data, and link clicks can help you understand where people are coming to your docs from, how long they’re staying on the page, how many readers there are, and what they’re doing after they get to a topic. Here are some example site metrics:

Page views
Session data like time on page
Referer data
Link clicks
Button clicks
Bounce rate
Client IP

You can also use this data to understand better how people interact with your documentation, like whether they’re using a version switcher on your page or expanding/collapsing more information hidden on the page.

You can also split this data by IP address to understand groups of topics that specific users are clustering around, to better understand how people use the documentation.

Identify questions based on your backlog

The process of adding data to your documentation prioritization strategy is all about making do with what you have to answer what you want to know. What you want to know depends on your backlog.

Data analysis is focused on a goal. You don’t want to collect a lot of data and then just stare at it, or get stressed out by the amount of “insights” that you could be gathering but meanwhile you’re not really sure what to do with the information. If you consider questions that you want to answer in advance, you can focus your data collection and analysis in a more valuable way.

Some example questions that you might identify based on your task list:

What are people looking for? Are they finding what they’re looking for?
Are people looking for information about /<thing I’ve been told to document/>?
What do people want more help with?
What people are we targeting that don’t see their use cases represented?

Tie questions to data types

After you’ve identified questions relevant to your task list, you can tie those questions to data types that can help you answer the questions.

For example, the question: What are people looking for and not finding?

To answer this, you can look where people are looking for information, namely search keywords that they’re typing into search engines, common questions being posted on forums, or the topics of support cases filed by customers.

For example, I looked at some data and was able to identify specific search terms people are using on the documentation site that are routing customers to a company-managed forum site. I can then use that data to identify cases where people are looking for documentation about something, but are not finding the answers in the documentation.

Another example question: What do people want more help with?

This could be answered by looking at the topics of support cases again, but also the types of questions being asked in training courses, as well as unanswered questions on forums.

As a final example: What market groups are we targeting that don’t see their use cases represented?

To answer this, you could look at data about sales leads, questions being asked by the field that contain specific use cases for various market verticals, as well as questions being asked in training courses.

Find questions from data

If you don’t have much of a task list to work with, or if you aren’t able to get access to data that can help you answer your questions, you can still make use of the data that is available to you and draw valuable insights from it.

You can identify interest in content that you maybe weren’t aware of, and make plans to write more to address that interest, or modify existing content to address that interest. Maybe there are a bunch of forum threads about how to do something, but nothing authoritative in the documentation. That information hasn’t made it to the docs writers in any way, but because you’re looking at the available data, you’re able to see that it’s important.

Even if you have no data specifically relevant to the documentation or customer questions, you can still find ways to identify documentation work to add to a task list. You could create datasets by performing text analysis on all or specific documentation topics, and identify complexity issues, or topics that don’t adhere to a style guide. You could use customer satisfaction surveys to identify places where documentation architecture or linking strategies could be improved.

Working with the data

Now you hopefully have a better understanding of different types of data available to you, and how you can identify valuable data sources based on your questions that you want to answer. But how much data do you need to collect? And how do you get the data? Most importantly, how do you analyze it to answer the questions you want to answer?

How much data?

How much data do you need to collect? You don’t need to collect data forever. You don’t need ALL the data. You just need enough data to point you in a direction and reduce uncertainty.

You can use a small sample of users, or a small sample of time, so long as it helps you answer your question and reduce uncertainty about what the answer could be. Collecting larger amounts of data doesn’t mean that you reduce uncertainty by an equally large degree. The amount of data you collect doesn’t correlate directly to what you’re able to learn from it. However, if the question you’re trying to answer with data concerns all the documentation users over a long period of time, you will be collecting more data than if you just want to know what a specific subset of readers found interesting on a Friday afternoon.

Try for representative samples that are relevant for the questions you’re trying to answer. If you can’t get representative data, try for a random sample. If you can’t get representative or random samples, acknowledge the bias that is inherent in the data you’re using. Add context to the data wherever possible, especially about who the data represents and why the data is still valuable if it isn’t representative.

You might find that collecting a small amount of data leaves you with more questions than answers, and that’s okay too. It’s an opportunity to continue exploring and learning more about your customers and your documentation tasks. But how do you even get any data at all?

How do you get the data?

You’ll either be collecting your own data, or asking others for the data you need.

If it’s data about the documentation site or its content, you might own that data yourself, and already have access to it. If it’s other types of data, like sales leads or user research data, it’s time to talk to the departments or people that manage those areas.

A business development department might have reporting on internal tools like sales leads or support cases.
Product managers can share direct customer data and product usage data if you don’t have direct access.
Project managers can share data related to internal development processes.

The teams managing different datasets will vary at your organization, and might even be you in many cases. They may be reluctant to share data. With that in mind, remember that when you collect data, you don’t need to get persistent access to all the data you want. Focus on getting some access to some data that is useful to answer your questions. After that, you can use that data to make your work more efficient and informed, and then hopefully communicate that value and get more access to data in the future if you want.

What to use for data analysis?

What do you use to analyze that data after you get it? How do you transform data into a report of useful information?

Some tools might already have analytics and reporting built in, like Google Analytics. That can certainly make it easier to analyze the data!

For other types of data that you need to analyze yourself, use the tools available to you. Think about what already know how to use, or have access to:

Know how to use Excel? Perfect! Get started collecting and processing data in spreadsheets and with macros.
Know how to write scripts in R/Python to analyze data? Great! You can write scripts to collect, process, and visualize this data.
Is your organization using a tool like Splunk, ElasticSearch, Tableau, etc.? Good news! You are really ready for data analysis.

You don’t have to spend a long time learning a new tool to analyze data for these purposes. If you continue incorporating data analysis into your work, it might make sense, but it isn’t necessary to get started.

Tools aren’t magic

It’s also important to note that tools aren’t magic. Some degree of data analysis will involve manual collecting, categorizing, or cleaning of the data. If your organization doesn’t have strict topic types, you might need to perform manual topic-typing. If you want to analyze some information but the data isn’t in a machine-readable format, you might have to sit at your desk copy pasting for hours.

Depending on your skills, the current state of the data that you want to analyze, and the tools available to you, the amount of time it takes to analyze data and get results can vary widely. I have spent 3 days manually processing data in Excel, and I’ve spent 2 hours creating searches in mostly-clean datasets in Splunk to get answers to various questions. Keep that in mind when you’re analyzing data.

How to perform data analysis

When you analyze data, what are you actually looking at?

Top, rare, outlying values

Find out what values are most common, and which values are least common. Those can be established by counting the various instances of values.

Look for values that are different from the others by a large margin. You can use standard deviation as a function to achieve this.

Patterns and clusters across data

You can also look for patterns and clusters in your data.

If you’re working with qualitative data, you might need to categorize, or code, the data so that you can sort it and look for patterns in the results. You can identify these patterns by counting instances of categories, or looking at clusters of behavior. An example of a cluster of behavior is if you look at documentation topic visits over time, and you identify a spike in visits at a particular time.

Split by different features

You also want to segment data by different features. Meaning, you can better understand the most common values if you split them by other types of information. For example, you can look at the most commonly visited topics in your documentation set over the last 3 months, or you can look at the most commonly visited topics in your documentation over the last 3 months, but on a week-to-week basis. That additional split can help you understand how those values are changing over time. If you identify a spike in a particular topic or category of topics, you can then interpret the data. Maybe a new product release led to a spike of interest in the release notes topic that wasn’t easily identified until you split the results by week. This is also a good opportunity to point out to a product team that people really do read your documentation!

That’s an example of splitting by time, but you can split by any other field available to you in your data. To use the same data type, looking at the most common topics by product, by IP address, or other factors, can help lead to valuable insights.

Combine data types

You can combine different types of data to understand approximately how many people are using the product vs how many of them are using the documentation. Comparing sales leads, product usage data, and existing page views could help you approximate the number of potential, and existing customers, alongside the number of distinct documentation readers.

Make sure that when you combine data across datasets, you keep track of units and time ranges, and make sure that you compare like data with like data. For example, be careful not to use data that refers to potential customers with data that refers to existing customers, because that could lead to misleading results if you don’t keep context with the data.

Interpreting results

When you interpret the results of your data analysis, make sure that you are adding context to the data. Especially when dealing with outlier data, but even when reviewing data like rarely-viewed or frequently-viewed topics, keep in mind additional context that could explain results.

Add context from expertise

Use your expertise and knowledge of the documentation to add context. For example, topics concerning a specific functionality are likely to be more popular at a specific time if that functionality was recently changed.

Pursue alternate explanations

Whenever you’re interpreting data, you want to make sure that you’re gut-checking it against what you already know. So if a relatively mundane topic has wildly out-of-the-ordinary page views, there are likely alternate explanations for that interest. Maybe your topic ended up being a great resource about cron syntax in general, even for people that don’t use your product.

Draw realistic conclusions

Draw realistic conclusions based on the data available to you. You might not be able to get access to or combine specific datasets due to privacy concerns. If you carefully identify what problems you’re trying to solve, and select only the data sources that can help you solve those problems, you can reduce the potential that you’ll introduce bias into your data analysis, and improve the conclusions that you’re able to draw.

Don’t trust data blindly

Don’t trust the data blindly. When reviewing data that seems out of the ordinary or like outliers, examine the different reasons why the data could be like that. Who does the data represent? What does it represent? Make sure that you’re interpreting data in context, so that you’re able to understand exactly what it represents. It can be tempting to ignore data that doesn’t match your biases or expectations.

Above all, remember to use data to complement your research and writing, and validate or challenge assumptions about your audience.

Your turn to add data

Identify the questions you’re trying to answer
Use the data available to you
Use the tools available to you
Analyze and interpret the data
Take action and prioritize accordingly

Additional resources

Find me on Twitter @smorewithface.
The Product is Docs book chapter on Measuring Success
How to Measure Anything book by Douglas Hubbard
Bob Watson’s posts on docsbydesign.com about measuring value
Just Enough Research book by Erika Hall
Nielsen Norman Group about handling small sample sizes.

The Concepts Behind the Book: How to Measure Anything

Sun, 19 May 2019 11:46:40 +0000

I just finished reading How to Measure Anything: Finding the Value of Intangibles in Business by Douglas Hubbard. It discusses fascinating concepts about measurement and observability, but they are tendrils that you must follow among mentions of Excel, statistical formulas, and somewhat dry consulting anecdotes. For those of you that might want to focus mainly on the concepts rather than the literal statistics and formulas behind implementing his framework, I wanted to share the concepts that resonated with me. If you want to read a more thorough summary, I recommend the summary on Less Wrong, also titled How to Measure Anything.

The premise of the book is that people undertake many business decisions and large projects with the idea that success of the decisions or projects can’t be measured, and thus they aren’t measured. It seems a large waste of money and effort if you can’t measure the success of such projects and decisions, and so he developed a consulting business and a framework, Applied Information Economics (AIE)to prove that you can measure such things. Near the end of his book on page 267, he summarizes his philosophy as six main points:

1. If it’s really that important, it’s something you can define. If it’s something you think exists at all, then it’s something that you’ve already observed somehow.

2. If it’s something important and something uncertain, then you have a cost of being wrong and a chance of being wrong.

3. You can quantify your current uncertainty with calibrated estimates.

4. You can compute the value of additional information by knowing the “threshold” of the measurement where it begins to make a difference compared to your existing uncertainty.

5. Once you know what it’s worth to measure something, you can put the measurement effort in context and decide on the effort it should take.

6. Knowing just a few methods for random sampling, controlled experiments, or even just improving on the judgment of experts can lead to a significant reduction in uncertainty.

To restate those points:

Define what you want to know. Consider ways that you or others have measured similar problems. What you want to know might be easier to see than you thought.
It’s valuable to measure things that you aren’t certain about if they are important to be certain about.
Make estimates about what you think will happen, and calibrate those estimates to understand just how uncertain you are about outcomes.
Determine a level of certainty that will help you feel more confident about a decision. Additionally, determine how much information will be needed to get you there.
Determine how much effort it might take to gather that information.
Understand that it probably takes less effort than you think to reduce uncertainty.

The crux of the book revolves around restating measurement from “answer a specific question” to “reduce uncertainty based on what you know today”.

Measure to reduce uncertainty

Before reading this book, I thought about data analysis as a way to find an answer to a question. I’d go in with a question, I’d find data, and thanks to that data, I’d magically know the answer. However, that approach only works with specifically-defined questions and perfect data. If I want to know “how many views did a specific documentation topic get last week” I can answer that straightforwardly with website metrics.

However, if I want to know “Was the guidance about how to perform a task more useful after I rewrote it?” there was really no way to know the answer to that question. Or so I thought.

Hubbard’s book makes the crucial distinction that data doesn’t need to exist to directly answer that question. It merely needs to make you more certain of the likely answer. You can make a guess about whether or not it was useful, carefully calibrating your guess based on your knowledge of similar scenarios, and then perform data analysis or measurement to improve the accuracy of your guess. If you’re not very certain of the answer, it doesn’t take much data or measurement to make you more certain, and thus increase your confidence in an outcome. However, the more certain you are, the more measurement you need to perform to increase your certainty.

Start by decomposing the problem

If you think what you want to measure isn’t measurable, Hubbard encourages you to think again, and decompose the problem. To use my example, and #1 on his list, I want to measure whether or not a documentation topic was more useful after I rewrote it. As he points out with his first point, the problem is likely more observable than I might think at first.

“Decompose the measurement so that it can be estimated from other measurements. Some of these elements may be easier to measure and sometimes the decomposition itself will have reduced uncertainty.”

I can decompose the question that I’m trying to answer, and consider how I might measure usefulness of a topic. Maybe something is more useful if it is viewed more often, or if people are sharing the link to the topic more frequently, or if there are qualitative comments in surveys or forums that refer to it. I can think about how I might tell someone that a topic is useful, what factors of the topic and information about it I might point to. Does it come up first when you search for a specific customer question? Maybe then search rankings for relevant keywords are an observable metric that could help me measure utility of a topic.

You can also perform extra research to think of ways to measure something.

“Consider your findings from secondary research: Look at how others measured similar issues. Even if their specific findings don’t relate to your measurement problem, is there anything you can salvage from the methods they used?”

Is it business critical to measure this?

Before I invest a lot of time and energy performing measurements, I want to make sure (to Hubbard’s second point in his list) that the question I am attempting to answer, what I am trying to measure, is important enough to merit measurement. This is also tied to points four, five, and six: does the importance of the knowledge outweigh the difficulty of the measurement? It often does, especially because (to his sixth point), the measurement is often easier to obtain than it might seem at first.

Estimate what you think you’ll measure

To Hubbard’s third point, a calibrated estimate is important when you do a measurement. I need to be able to estimate what “success” might look like, and what reasonable bounds of success I might expect are.

Make estimates about what you think will happen, and calibrate those estimates to understand just how uncertain you are about outcomes.

To continue with my question about a rewritten topic’s usefulness, let’s say that I’ve determined that added page views, elevated search rankings, and link shares on social media will mean the project is a success. I’d then want to estimate what number of each of those measurements might be meaningful.

To use page views as an example for estimation, If page views increase by 1%, it might not be meaningful. But maybe 5% is a meaningful increase? I can use that as a lower bound for my estimate. I can also think about a likely upper bound. A 1000% increase would be unreasonable, but maybe I could hope that page views would double, and I’d see a 100% increase in page views! I can use that as an upper bound. By considering and dismissing the 1% and 1000% numbers, I’m also doing some calibration of my estimates—essentially gut checking them with my expertise and existing knowledge. The summary of How to Measure Anything that I linked in the first paragraph addresses calibration of estimates in more detail, as does the book itself!

After I’ve settled on a range of measurement outcomes, I can assess how confident I am that this might happen. Hubbard calls this a Confidence Interval. I might only be 60% certain that page views will increase by at least 5% but they won’t increase more than 100%. This gives me a lot of uncertainty to reduce when I start measuring page views.

One way to start reducing my uncertainty about these percentage increases might be to look at the past page views of this topic, to try to understand what regular fluctuation in page views might be over time. I can look at the past 3 months, week by week, and might discover that 5% is too low to be meaningful, and a more reasonable signifier of success would be a 10% or higher increase in page views.

Estimating gives me a number that I am attempting to reduce uncertainty about, and performing that initial historical measurement can already help me reduce some uncertainty. Now I can be 100% certain that a successful change to the topic should show more than 5% page views on a week-to-week basis, and maybe am 80% certain that a successful change would show 10% or more page views.

When doing this, keep in mind another point of Hubbards:

“a persistent misconception is that unless a measurement meets an arbitrary standard….it has no value….what really makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.”

If you’re choosing to undertake a large-scale project that will cost quite a bit if you get it wrong, you likely want to know in advance how to measure the success of that project. This point also underscores his continued emphasis on reducing uncertainty.

For my (admittedly mild) example, it isn’t valuable for me to declare that I can’t learn anything from page view data unless 3 months have passed. I can likely reduce uncertainty enough with two weeks of data to learn something valuable, especially if my uncertainty level is in relatively low (in this example, in the 40-70% range).

Measure just enough, not a lot

Hubbard talks about the notion of a Rule of Five:

There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population.

Knowing the median value of a population can go a long way in reducing uncertainty. Even if you can only get a seemingly-tiny sample of data, this rule of five makes it clear that even that small sample can be incredibly valuable for reducing uncertainty about a likely value. You don’t have to know all of something to know something important about it.

Do something with what you’ve learned

After you perform measurements or do some data analysis and reduce your uncertainty, then it’s time to do something with what you’ve learned. Given my example, maybe my rewrite increased page views of the topic by 20%, something I’m now fairly certain is a significant degree, and it is now higher in the search results. I’ve now sufficiently reduced my uncertainty about whether or not the changes made this topic more useful, and I can now rewrite similar topics to use a similar content pattern with confidence. Or at least, more confidence than I had before.

Overall summary

My super abbreviated summary of the book would then be to do the following:

Start by decomposing the problem
Ask is it business critical to measure this?
Estimate what you think you’ll measure
Measure just enough, not a lot
Do something with what you’ve learned

I recommend the book (with judicious skimming), especially if you need some conceptual discussion to help you unravel how best to measure a specific problem. As I read the book, I took numerous notes about how I might be able to measure something like support case deflection with documentation, or how to prioritize new features for product development (or documentation). I also considered how customers might better be able to identify valuable data sources for measuring security posture or other events in their data if they followed many of the practices outlined in this book.

Not sober curious, just sober

Thu, 14 Mar 2019 08:15:07 +0000

An article covering the “Sober Curious Movement” was published in the Chicago Tribune a few weeks ago. My brother shared it with me, and I’m still thinking about it. The article discusses a “sober curious” movement in America and interviews a number of people in Chicago that have chosen to quit drinking. Apparently because they quit drinking for different reasons than alcoholism or binge drinking, they are “sober curious” instead of simply “sober”. (It’s a book too).

My brother sent it to me because I don’t drink either, which can feel like an oddity in your twenties. I quit drinking at concerts when I was 22, after I went to a concert, had one drink, and ended up fainting in between the opener and the headliner. Thinking it was just a fluke of that night, I tried again at another show a few months later, and spent the headlining set sitting down in the back of the venue to avoid fainting a second time. After that I realized that it wasn’t worth it, and never drank at a concert again.

It took longer for me to quit drinking overall, and I’d make exceptions at time for special occasions when it just felt too awkward to not drink—weddings, parties, first dates—but after awhile I decided to stop making the exceptions. It was part personal challenge, and part health-conscious decision. My body had never responded well to alcohol, what with lightheadedness or nausea following anything more than a couple drinks. By the time I was 22 I had a short list of “okay alcohols” and quantities, and by the time I was 26 I’d grown tired of bothering.

My life had shifted to involve fun activities beyond drinking, and my friends weren’t drinking-focused either. I’d be going to concerts or to the gym/the soccer pitch every other day, and drinking just didn’t fit anywhere. I’d spent time in college not drinking at various parties, where I knew I had to be fresh for studying the next day, so I knew I could still have fun without drinking. Choosing to quit drinking overall felt like a natural progression.

So where does that leave me now, and why am I still so peeved at that article? For one, it only quotes women. I like to see women quoted by journalists, but by only quoting women, the choice to be sober felt somewhat trivialized. In addition, the women’s comments were contextualized with talk of mindfulness and yoga, as though this is a choice being made by a particular type and class of woman, and no others. It also perpetuates the notion that having fun without drinking is some strange novelty. There are a lot of people out there that have fun without drinking. Indeed, as one of the women in the article points out—it’s a challenge to your confidence to go out and be all of yourself, without alcohol. But it can be that much more invigorating in that way. You get to challenge your social anxiety and actively build confidence, rather than relying on alcohol and wondering if you can talk to strangers without it at all.

I think there’s also harm in talking about sobriety distinct from alcoholism. A lot of quitting drinking is about realizing that you don’t like who you are when you drink (or after you drink). The recent essay about the Joe Beef restaurateurs makes this clear. Some people have the lifestyles, genetic predisposition, or experienced traumas that escalate their alcohol consumption to recognizable alcoholism. Others have a dependence on it that they dislike, even if others don’t see it as an issue (as one of the women interviewed in the Chicago Tribune article mentions). It’s more than okay to share that common understanding, rather than separate ourselves into different groups, “the sober curious” and “the fully sober due to addiction”. That’s harmful. Indeed, the sober curious meetup in Chicago also includes “young women in recovery”. They get it.

Often, quitting drinking feels like a social choice more than a personal choice. It feels that way largely due to the fact that there often aren’t that many sober social activities out there. It’s hard to stay out late with friends sober when the only places open late are bars. It’s harder to choose yourself over alcohol, when it can often mean isolating yourself from friends. So while I struggle with the rhetoric and the patronizing presentation of the “sober curious” movement, I absolutely support it as an overall societal direction. Here’s to more late-night diners, pastry places like Mission Pie, and sober pop-ups like Brillig Dry Bar that help us sober people stay up late and out with friends.

Planning and analyzing my concert attendance with Splunk

Sun, 10 Mar 2019 22:23:46 +0000

This past year I added some additional datasets to the Splunk environment I use to analyze my music: information about tickets that I’ve purchased, and information about upcoming concerts.

Ticket purchase analysis

I started keeping track of the tickets that I’ve purchased over the years, which gave me good insights about ticket fees associated with specific ticket sites and concert promoters.

Based on the data that I’ve accumulated so far, Ticketmaster doesn’t have the highest fees for concert tickets. Instead, Live Nation does. This distinction is relatively meaningless when you realize they’ve been the same company since 2010.

However, the ticket site isn’t the strongest indicator of fees, so I decided to split the data further by promoter to identify if specific promoters had higher fees than others.

Based on that data you can see that the one show I went to promoted by AT&T had fee percentages of nearly 37%, and that shows promoted by Live Nation (through their evolution and purchase by Ticketmaster) also had fees around 26%. Shows promoted by independent venues have somewhat higher fees than others, hovering around 25% for 1015 Folsom and Mezzanine, but shows promoted by organizations whose only purpose is promotion tend to have slightly lower fees, such as select entertainment with 18%, Popscene with 16.67%, and KC Turner Presents with 15.57%.

I realized I might want to refine this, so I recalculated this data, limiting it to promoters from which I’ve bought at least two tickets.

It’s a much more even spread in this case, ranging from 25% to 11% in fees. However, you can see that the same patterns exist— for the shows I’ve bought tickets to, the independent venues average 22-25% in fees, while dedicated independent promoters are 16% or less in added fees, with corporate promoters like Another Planet, JAM, and Goldenvoice filling the middle of the data ranging from 18% to 22%.

I also attempted to determine how I’m discovering concerts. This data is entirely reliant on my memory, with no other data to back it up, but it’s pretty fascinating to track.

It’s clear that Songkick has become a vital service in my concert-going planning, helping me discover 46 shows, and friends and email newsletters from venues helping me stay in the know as well for 19 and 14 shows respectively. Social media contributes as well, with a Facebook community (raptors) and Instagram making appearances with 10 and 2 discoveries respectively.

Concert data from Songkick

Because Songkick is so vital to my concert discovery, I wanted to amplify the information I get from the service. In addition to tracking artists on the site, I wanted to proactively gather information about artists coming to the SF Bay Area and compare that with my listening habits. To do this, I wrote a Songkick alert action in Python to run in Splunk.

Songkick does an excellent job for the artists that I’m already tracking, but there are some artists that I might have just recently discovered but am not yet tracking. To reduce the likelihood of missing fast-approaching concerts for these newly-discovered artists, I set up an alert to look for concerts for artists that I’ve discovered this year and have listened to at least 5 times.

To make sure I’m also catching other artists I care about, I use another alert to call the Songkick API for every artist that is above a calculated threshold. That threshold is based on the average listens for all artists that I’ve seen live, so this search helps me catch approaching concerts for my historical favorite artists.

Also to be honest, I also did this largely so that I could learn how to write an alert action in Splunk software. Alert actions are essentially bits of custom python code that you can dispatch with the results of a search in Splunk. The two alert examples I gave are both saved searches that run every day and update an index. I built a dashboard to visualize the results.

I wanted to use log data to confirm which artists were being sent to Songkick with my API request, even if no events were returned. To do this I added a logging statement in my Python code for the alert action, and then visualized the log statements (with the help of a lookup to match the artist_mbid with the artist name) to display the artists that had no upcoming concerts at all, or had no SF concerts.

For those artists without concerts in the San Francisco Bay Area, I wanted to know where they were going instead, so that I could identify possible travel locations for the future.

It seems like Paris is the place to be for several of these artists—there might be a festival that LAUER, Max Cooper, George Fitzgerald, and Gerald Toto are all playing at, or they just happen to all be visiting that city on their tours.

I’m planning to publish a more detailed blog post about the alert action code in the future on the Splunk blogs site, but until then I’ll be off looking up concert tickets to these upcoming shows….

Engaging with San Francisco history as a newcomer

Thu, 07 Mar 2019 21:17:25 +0000

I moved to San Francisco from the Midwest a few years ago, and I’d been missing a strong sense of history since then. I’ve been to a few events in an attempt to learn more about my new home, such as a Dolores Park history day, or a history-relevant event in the Mission as part of Litcrawl, but I struggled to absorb a history for the city that went beyond “gold rush, earthquake, tech boom, bust, boom”.

But last week I was at the library and saw an event that was being held in conjunction with the display of the Bay Model throughout SF public libraries and Take Part SF, called Vanished Waters. It was about Mission Bay history so I skipped my regular workout to attend. It was well worth it!

Vanished Waters is also a book, so that was the loose structure that the talk revolved around, and was given by Chris Carlsson, an expert on San Francisco history and an engaging speaker. He co-founded Shaping SF, helping to maintain a digital archive of the city’s past.

My favorite fascinating facts that I learned at the talk were that in 1852, Market street ended where 3rd street is in an 80 ft tall sand dune.

SoMA was really hilly and marshy, but then some dude with a steam shovel was like “sup let me move that sand for you” and also “sup let me help you fill in this lot that you bought that is literally just water”. That’s my paraphrasing, but the actual details are to be read on Found SF.

The whole idea to fill the San Francisco Bay in is hard to imagine now because it’s not polluted, but if it was a stinky polluted putrid mess full of garbage it’s easier to imagine it being a good idea. (The whole idea to fill in the Bay is why the SF Bay Model was built).

However, my favorite part of the talk was when Carlsson discussed using this history to inform our present and future decisions. He pointed out that there is a lot of rhetoric in San Francisco about how to build more housing to manage the growth of the city, and what kinds of development is best suited to accommodating all of the people that move here.

However, there’s not much rhetoric (if any) about staging a managed retreat from climate change. San Francisco is a coastal city, built on top of marshland, sand dunes, or literal land fill. What happens when the sea level begins to rise, or more volatile weather patterns cause bigger storms and potential flooding?

I realized after this talk that some city dwellers love to judge southeastern coastal city residents that build or rebuild homes in the path of hurricanes or immediate climate change threats, and yet, New York City and San Francisco are both at high risk from sea level rise.

That’s not to mention the earthquake risk in San Francisco. Our current development plans are not necessarily smart, as this article in the New York Times points out.

It’s fascinating to explore what the city used to look like less than 200 years ago, and imagine what it might look like in 2057 in the face of climate change. I lost nearly an hour clicking around the maps on David Rumsey’s website (recommended by Carlsson).

Here’s a map of the 1857 coast, overlaid on modern San Francisco.

This map in 1869 of land lots makes it clear just how much of the land that was sold during that period wasn’t actually land. This essay in Collector’s Weekly covers how that land speculation happened and how it shapes modern real estate in the city.

This exploration all happened because of this talk and the display of the 1938 3D model of San Francisco in the San Francisco Public Libraries. If you want to help find the model a permanent home for display in the city, sign this petition. Just imagine making this map overlay of what San Francisco looked like in 1938 into a tactile experience.

The model is still on display in SF Public Library branches throughout the city, and you can stay engaged in city history through the San Francisco Department of Memory, the California Historical Society, Shaping SF, and Found SF.

Data enrichment at ingest-time, not search time, with Cribl

Wed, 06 Mar 2019 20:21:30 +0000

Disclaimer: I’m a Splunk employee, and I’m not a Cribl customer, but I do know the founders (including the author of the blog post). I figured I’d write this exploration up here rather than as an exceedingly-long Twitter thread. My reactions are all to the content of the blog post, not actual use of the product.

If I’m reading this blog post from Cribl correctly, their product makes it easy to enrich events with metadata at ingest-time. This is relevant/exciting for me because when I’m ingesting music data for my side project, I’m only ever getting the initial slice that’s available from a specific REST endpoint or in a file.

I’ve been identifying and collecting additional data sources that I want to enrich my dataset with, but doing so requires extra calls to other endpoints in the same API, or other web services, which means I then need to figure out where I want to store all of that data.

It quickly turns into an architectural and conceptual headache that I delay handling, because I know I’d either be dumping a lot of data into lookups / the KV store, or having to seriously level up my Python skills and do data processing and enrichment in my code before sending it to Splunk Enterprise.

As a specific example, I use the Last.fm getRecentTracks endpoint to send my listening data to Splunk Enterprise, but to enrich that data with additional metadata like track duration, or album release date, I’d have to hit 2 additional endpoints (track.getInfo and album.getInfo, respectively).

Deciding when in the data processing pipeline to hit those endpoints, how to hit them, and where to store that information to enrich my events has been a struggle that I’ve been avoiding dealing with.

There is an advantage to collecting the metadata once and storing it in a lookup or the KV store, because the metadata is relatively static. That means that it is relatively straightforward to call an endpoint, collect the data, and store it somewhere for when I need it. That means that I then have the added flexibility to enrich my events with extra data at search time when I want to, but not otherwise.

However, this means that I’m having to make conceptual decisions at multiple points—when collecting the data, when deciding what format to store it in, and where, and when I am enriching events at search time. It’s a lot of added complexity, but this type of enrichment doesn’t affect the size of my originally-indexed events, though it might end up being indexed separately instead.

But with Cribl’s solution, I’d be making that choice once. That does mean I lose potential flexibility about when and which events I can enrich with the data, but it also means that the conceptual decisions aren’t something I have to belabor. I can enrich my listening data at ingest-time with additional metadata about the album, artist, and track, then send it on to be indexed. Then when I’m searching and want to perform additional work with the metadata, it’s all right there with my events already.

This is a convenient, if imperfect, solution for my use case. But my use case is pretty basic: enrich events with static information that might be shared across many events. That’s a use case with a lot of potential solutions. I could use this approach if I didn’t care about reducing the amount of data that I indexed to the bare minimum, and focused instead on convenience and context for my data ingestion, allowing me to save time when searching my data.

This solution is much more exciting for use cases other than mine, where you’re enriching events with dynamic information that is relevant and true for specific events at index-time. The blog post includes an example of this, combining the web access logs with context from proxy logs, making the time-to-discovery for investigations that use web access logs shorter.

There is flexibility in combining data at search time, but there is complexity with that approach as well. Cribl shows that there is convenience in creating that context at index-time as well.

Making Concert Decisions with Splunk

Sat, 23 Feb 2019 19:35:30 +0000

The annual Noise Pop music festival starts this week, and I purchased a badge this year, which means I get to go to any show that’s a part of the festival without buying a dedicated ticket.

That means I have a lot of choices to make this week! I decided to use data to assess (and validate) some of the harder choices I needed to make, so I built a dashboard, “Who Should I See?” to help me out.

First off, the Wednesday night show. Albert Hammond, Jr. of the Strokes is playing, but more people are talking about the Baths show the same night. Maybe I should go see Baths instead?

If I’m making my decisions purely based on listen count, it’s clear that I’m making the right choice to see Albert Hammond, Jr. It is telling, though, that I’ve listened to Baths more recently than him, which might have contributed to my indecision.

The other night I’m having a tough time deciding about is Saturday night. Beirut is playing, but across the Bay in Oakland. Two other interesting artists are playing closer to home, Bob Mould and River Whyless. I wouldn’t normally care about this so much, but I know my Friday night shows will keep me busy and leave me pretty tired. So which artist should I go see?

It’s pretty clear that I’m making the right choice to go see Beirut, especially given my recent renewed interest thanks to their new album.

I also wanted to be able to consider if I should see a band at all! This isn’t as relevant this week thanks to the Noise Pop badge, but it currently evaluates if the number of listens I have for an artist exceeds the threshold that I calculate based on the total number of listens for all artists that I’ve seen live in concert. To do this, I’m evaluating whether or not an artist has more listens than the threshold. If they do, I return advice to “Go to the concert!” but if they don’t, I recommend “Only if it’s cheap, yo.”

Because I don’t need to make this decision for Noise Pop artists, I picked a few that I’ve been wanting to see lately: Lane 8, Luttrell, and The Rapture.

While my interest in Lane 8 has spiked recently, there still aren’t enough cumulative listens to put them over the threshold. Same for Luttrell. However, The Rapture has enough to put me over the threshold (likely due to the fact that I’ve been listening to them for over 10 years), so I should go to the concert! I’m going to see The Rapture in May, so I am gleefully obeying my eval statement!

On a more digressive note, it’s clear to me that this evaluation needs some refinement to actually reflect my true concert-going sentiments. Currently, the threshold averages all the listens for all artists that I’ve seen live. It doesn’t restrict that average to consider only the listens that occur before seeing an artist live, which might make it more accurate. That calculation would also be fairly complex, given that it would need to account for artists that I’ve seen multiple times.

However, number of listens over time doesn’t alone reflect interest in going to a concert. It might be useful to also consider time spent listening, beyond count of listens for an artist. This is especially relevant when considering electronic music, or DJ sets, because I might only have 4 listen counts for an artist, but if that comprises 8 hours of DJ sets by that artist that I’ve listened to, that is a pretty strong signal that I would likely enjoy seeing that artist perform live.

I thought that I’d need to get direct access to the MusicBrainz database in order to get metadata like that, but it turns out that the Last.fm API makes some available through their track.getInfo endpoint, so I just found a new project! In the meantime I am able to at least calculate duration for tracks that exist in my iTunes library.

I now have a new avenue to explore with this project, collecting that data and refining this calculation. Reach out on Twitter to let me know what you might consider adding to this calculation to craft a data-driven concert-going decision-making dashboard.

If you’re interested in this app, it is open sourced and available on Splunkbase. I’ll commit the new dashboard to the app repo soon!

So you want to be a technical writer

Tue, 05 Feb 2019 17:12:59 +0000

If you’re interested in becoming a technical writer, or are new to the field and want to deepen your skills and awareness of the field, this blog post is for you.

What do technical writers actually do?

Technical writers can do a lot of different things! People in technical writing write how-to documentation, craft API reference documentation, create tutorials, even provide user-facing text strings to engineers.

Ultimately, technical writers:

Research to learn more about what they are documenting.
Perform testing to verify that their documentation is accurate and validate assumptions about the product.
Write words that help readers achieve specific learning objectives and that capture what the writer has learned in the research and testing processes.
Initiate reviews with engineers, product managers, user experience designers, quality assurance testers, and others to validate the accuracy, relevancy, and utility of the content.
Advocate for the customer or whoever uses the product or service being documented.

The people reading what technical writers have produced could be using software they’ve purchased from your company, evaluating a product or service they are considering purchasing, undergoing a required process controlled by your organization, writing code that interfaces with your services, configuring or installing modifying hardware produced by your company, or even reviewing the documentation for compliance and certification purposes. Your goal, if you choose to accept it, is to help them get the information they need and get back to work as soon as possible.

Identify what you want from your career

Some general career-assessment tips:

Identify what motivates you and what challenges you.
Identify what type of team environment you want. These are loose descriptions of types of team environments that are out there:
- A large highly-collaborative team with lots of interaction
- A distributed team that is available for questions and brainstorming as needed, but largely everyone is working on their own thing.
- A small team that collaborates as needed.
- A team of one, it’s just you, you are the team.

Is technical writing a good fit for you?

Do you enjoy explaining things to other people?
Do people frequently ask you to help explain something to them?
Do people frequently ask you to help them revise content for them?
Do you care or enjoy thinking about how to communicate information?
Do you identify when things are inconsistent or unclear and ask people to fix it? (Such as in a UI implementation, or when reviewing a pull request)
Do you enjoy problem-solving and communication?
Do you like synthesizing information from disparate sources, from people to product to code to internal documentation?
Do you enjoy writing?

My background and introduction to technical writing

I started in technical support. In college I worked in desktop support for the university, wandering around campus or in the IT shop, repairing printers, recovering data from dying hard drives, running virus scans, and updating software. After graduation I eventually found a temp job working phone support with University of Michigan, managing to turn that position into a full-time permanent role and taking on two different queues of calls and emails.

However, after a year I realized that was super exhausting to me. I couldn’t handle being “on” all day, and I found myself enjoying writing the knowledge base articles that would record solutions for common customer calls. I wrote fifty of them by the time I discovered a posting for an associate-level documentation specialist.

I managed to get that position, and transferred over to work with a fantastic mentor that taught me a ton about writing and communicating. After a few years in that position, writing everything from communication plans (and the accompanying communications), technical documentation, as well as a couple video scripts, I chose to move to California.

With that came another set of job hunting, and realizing that there are a lot of different job titles that technical writing can fall under: UI writer, UI copywriter, technical writer, documentation specialist, information developer… I set up job alerts, and ended up applying, interviewing, and accepting an offer for a technical writing position at Splunk. I’ve been at Splunk for several years now, and recently returned to the documentation team after spending nearly a year working in product management.

Where people commonly go to technical writing from

Technical writers can get their start anywhere! Some people become technical writers right out of college, but others transition to it after their career has already begun.

As a technical writer, your college degrees doesn’t need to be in technical writing, or even a technical-specific or writing-specific field. I studied international studies, and I’ve worked with colleagues that have studied astronomy, music, or statistics. Others have computer science or technical communication degrees, but it’s not a requirement.

For people transitioning from other careers, here are some common starting careers:

Software developers
UX practitioners
Technical support

That’s obviously a short list, but again if you care about the user and communication in your current role, that background will help you immensely in a technical writing position.

Prepare for a technical writing interview

Prepare a portfolio of writing samples

Every hiring manager wants to see a collection of writing samples that demonstrate how you write. If you don’t work in technical writing yet, you might not have any. Instead, you can use:

Contributions you’ve made to open source project documentation. For example, commits to update a README: https://github.com/yahoo/gryffin/pull/1
How-to processes you’ve written. For example, instructions for performing a code review or a design review.
A blog post about a technical topic that you are familiar with. For example, a post about a newly-discovered functionality in CSS.
Basic task documentation about software that you use. For example, write up a sample task for how to create a greeting card in Hallmark Card Studio.

Your portfolio of writing samples demonstrates to hiring managers that you have writing skills, but also that you consider how you organize content, how you write for a specific audience, and the level of detail that you include based on that audience. The samples that you use don’t have to be hosted on a personal website and branded accordingly. The important thing is to have something to show to hiring managers.

Depending on the interviewer, you might perform a writing exercise in-person or as part of the screening process. If you don’t have examples of writing like this, that’s a good reason to track down some open source projects in need of some documentation assistance!

Learn about the organization and documentation

Going in to the interview, make sure you are familiar with the organization and its documentation.

Read up about the organization or company that you are interviewing with. If you can, track down a mission statement for the organization.
Find the different types of documentation available online, if possible, and read through it to get a feel for what the team might be publishing.
If the organization provides a service or product that you’re able to start using right away, do that!

All of these steps help you better understand how the organization works, what the team you might be working on is producing, and demonstrates to the interviewer that you are motivated to understand what the role and the organization are about. Not to mention, this makes it clear that you have some of the necessary skills a technical writer needs when it comes to information-gathering.

Questions you might want to ask

Find out some basic team characteristics:

How many other technical writers are at the organization?
What org are the technical writers part of?
Is there a central documentation team or are the writers scattered across the organization?
How distributed is the documentation team and/or the employees at the organization?

Learn about the documentation process and structure:

What does the information-development process look like for the documentation? Does it follow semi-Agile methods and get written and researched as part of the development team, or does information creation follow a more waterfall style, where writers are delivered a finished product and expected to document it? Or is it something else entirely?
Are there editors or a style guide?
Do the writers work directly with the teams developing the product or service?
What sort of content management system (CMS) is in use? Is it structured authoring? A static-site generator reliant on documentation files written in markdown stored next to the code? A wiki? Something else?

Find out how valuable documentation is to the organization:

Do engineers consider documentation vital to the success of the product or service?
Do product managers?
Do you get customer feedback about your documentation?
What is the goal of documentation for the organization?

Some resources for getting started with technical writing

Books to read

These books cover technical writing principles, as well as user design principles. None of these links are affiliate links, and the proceeds of the book I helped author go to charity.

The Product is Docs by Christopher Gales and the Splunk documentation team
- Yes, I helped.
Every Page is Page One by Mark Baker
- This book is a great introduction and framework for writing documentation for the web.
Developing Quality Technical Information by Michelle Carey, Moira McFadden Lanyi, Deirdre Longo, Eric Radzinski, Shannon Rouiller, and Elizabeth Wilde.
- This book is a great resource and reference for detailed writing guidance, as well as information architecture.
Design of Everyday Things by Don Norman
- The classic design book covers user-focused principles that are crucial to writing good documentation.

This is an intentionally short list featuring books I’ve found especially useful. You can also consider reading Scenario-Focused Engineering: A toolbox for innovation and customer-centricity, Nicely Said: Writing for the Web with Style and Purpose, Content Everywhere: Strategy and Structure for Future-Ready Content, Design for How People Learn, and Made to Stick: Why Some Ideas Survive and Others Die.

Articles and blogs about technical writing

I like following resources in RSS feeds to get introduced to good thinking about technical writing, but not all good content is new content! Some great articles that have helped me a lot:

Blogs to follow (intermittently updated)

Beth Aitman’s http://uiwriting.tumblr.com/
Mark Baker’s http://everypageispageone.com/
Tom Johnson’s http://idratherbewriting.com/
Nielsen Norman Group articles: https://www.nngroup.com/articles/

Great articles about technical writing

How Chunking Helps Content Processing by Kate Moran for Nielsen Norman Group
Writing great documentation by Taylor Singletary
Users’ Advocate: Balancing Just-in-Time Support Docs and Customer Experience by Neal Kaplan (who I now work with)
How can technical writers cut through engineering jargon and decode complex information? by Tom Johnson
Top 10 Most Frequently Asked Questions about Technical Writing by Tom Johnson
A little thing about release notes by the Slack team
How to write a great error message by Thomas Fuchs
What is the technical writer’s role in content marketing? by Tom Johnson
Writing truthful documentation by Tom MacWright
Inverted Pyramid: Writing for Comprehension by Amy Schade for Nielsen Norman Group
Plain Language Is for Everyone, Even Experts by Hoa Loranger for Nielsen Norman Group
Break Grammar Rules on Websites for Clarity by Hoa Loranger for Nielsen Norman Group

Other web resources

Twitter is a great resource for building a network of people that care about documentation. If you use it, I recommend searching for people who commonly tweet with #writethedocs.

Write the Docs is a conference and community founded by Eric Holscher and maintained by a brilliant set of volunteers!

The Write the Docs Slack workspace is fairly active, and includes channels for job postings, career advice, as well as current discussions about trends and challenges in the technical writing world.

Some talks from the conference I recommend checking out are visible on YouTube:

Write the Docs Portland 2017: Error Messages: Being Humble, Human, and Helpful… by Kate Voss
Write the Docs Portland 2017: Caring Systems: Documentation as care by Amelia Abreu
Write the Docs Portland 2017: Even Naming This Talk Is Hard by Ruthie BenDor
Write the Docs Portland 2017: Start with the tasks, not the endpoints by Sarah Hersh
Write the Docs Portland 2016: Write the Readable README by Daniel Beck
Write the Docs Portland 2016: Copy That: Helping your Users Succeed with Effective Product Copy by Sarah Day
Write the Docs Portland 2016: 7 Values of Effective Tech Writing Teams by Joao Fernandes
Write the Docs Portland 2016: Oops, I Became an Engineer by Tara Scherner de la Fuente

There are playlists for 2018 (which I did not attend) and earlier years as well on YouTube, so dig around there and find some more resources too if watching videos is useful to you!

My 2018 Year in Music: Data Analysis and Insights

Sun, 09 Dec 2018 16:27:39 +0000

This past year has been pretty eventful in music for me. I’ve attended a couple new festivals, seen shows while traveling, and discovered plenty of new bands. I want to examine the data available to me and contrast it with my memories of the past year.

I’ve been using Splunk to analyze my music data for the past couple years. You can learn more about what I’ve learned from that in the past in my other posts, see Reflecting on a Decade of Quantified Music Listening and Best of 2017: Newly-Discovered Music. I also wrote a blog post for the Splunk blog (I work there) about this too: 10 Years of Listens: Analyzing My Music Data with Splunk.

Comparing Spotify’s Data with Mine

Spotify released its #2018wrapped campaign recently, sharing highlights from the year of my listening data with me (and in an ad campaign, aggregate data from all the users). As someone that uses Spotify but not as my exclusive source of music listening, I was curious to compare the results with my holistic dataset that I’ve compiled in Splunk.

Spotify’s top artists for me were somewhat different from the results that I found from the data I gather from Last.fm and analyze with Splunk software. Spotify and my holistic listening data agree that I listened to Poolside more than anyone else, and was also a big fan of Born Ruffians, but beyond that they differ. This is probably due to the fact that I bought music and when I’m mobile I switch my primary listening out of Spotify to song files stored on my phone.

In addition, my top 5 songs of the year were completely different from those listed in Spotify. My holistic top 5 songs of the year were all songs that I purchased. I don’t listen to music exclusively in Spotify, and my favorites go beyond what the service can recognize.

Spotify identified that I’ve listened to 30,473 minutes of music, but I can’t make a similarly reliable calculation with my existing data because I don’t have track length data for all the music that I’ve listened to. I can calculate the number of track listens so far this year, and based on that, make an approximation based on the track length data that I do have from my iTunes library. The minute calculation I can make indicates that I’ve so far spent 21,577 minutes listening to 3,878 of the 10,301 total listens I’ve accumulated so far this year (Numbers to change literally as this post is being written).

I’m similarly lacking data allowing me to determine my top genre of the year, but Indie is a pretty reliable genre for my taste.

Other Insights from 2018

I was able to calculate my Top 10 artists, songs, and albums of the year, and drill down on the top 10 artists to see additional data about them (if it existed) in my iTunes library, like other tracks, the date it was added, as well as the kind of file (helping me identify if it was purchased or not), and the length of the track.

There are quite a few common threads across the top 10 artists, songs, and albums, with Poolside, Young Fathers, Gilligan Moss, The Vaccines, and Justice making consistent appearances. The top 10 songs display obsessions with particular songs that outweigh an aggregate popularity for the entire album, leading other songs to be the top albums of the year.

Interestingly, the Polo & Pan album makes my top 10 albums while they don’t make it to my top 10 artist or song lists. This is also true for the album Dancehall by The Blaze. I’m not much of an album listener usually, but I know I listened to those albums several times.

The top 10 song list is more dominated by specific songs that caught my attention, and the top 10 artists neatly reflect both lists. The artists that have a bit more of a back catalog also reveal themselves, given that Born Ruffians managed to crack the top 10 despite not having any songs or albums make the top 10 lists, and Hey Rosetta! makes the top artist and album lists, despite having no top songs.

I purchased 285 songs this year, an increase of 157 compared to the year before. I think I just bought songs more quickly after first hearing them this year, and there are even some songs missing from this list that I bought on Beatport or Bandcamp because they weren’t available in the iTunes Store. While I caved in to Spotify premium this year, I still kept up an old promise to myself to buy music (rather than acquire it without paying for it, from a library or questionable download mechanisms) now that I can afford it.

A Year of Concerts

I’ve been to a lot of concerts so far this year. 48, to be exact. I spent a lot of money on concert tickets, both for the shows I attended this year and for shows that went on sale during 2018 (but at this point, might be happening in 2019). I often will buy tickets for multiple people, so this number isn’t very precise for my own personal ticket usage.

I managed to go to at least 2 concerts every month. By the time the year is over, I’m on track to go to 51 different shows. Based on the statistics, there are some months where I went to many more than 1 show per week, and others where I didn’t. Especially apparent are the months with festivals—February, August, and October all included festivals that I attended.

Many of those festivals brought me to new-to-me locations, with the Noise Pop Block Party and Golden Gate Park giving me new perspectives on familiar places, and Lollapalooza after shows bringing me out to Schubas Tavern for the first time in Chicago.

If you’re reading this wondering what San Francisco Belle is, it’s a boat. That’s one of several new venues that electronic music brought me to—DJ sets on that boat as part of Goldroom and Gigamesh’s tour, plus a day party in Bergerac and a nighttime set at Audio other times throughout the year.

Some of those new venue locations brought newly-discovered music to me as well.

The 20th-most-popular artist I discovered this year was Jenn Champion, who opened for We Were Promised Jetpacks at their show at the Great American Music Hall. I started writing this assuming that I hadn’t heard Jenn Champion before that night, but apparently I first discovered them on July 9, but the show wasn’t until October 9.

As it turns out, I listened to what is now my favorite song by Jenn Champion that day in July, likely as part of a Spotify algorithm-driven playlist (judging by the listening neighbors around the same time) but it didn’t stick until I saw them play live months later. The vagaries of playlists that refresh once a week can mean fleeting discoveries that you don’t really absorb.

Other songs I listened to that day in July

Because of how I can search for things in Splunk, I was also curious to see what others songs I heard when I first discovered Hubert Kirchner, a great house artist.

I have really no idea what playlist I was listening to that might have led to me making jumps from Sofi Tukker, to Tanlines, to Dion, to Deradoorian, then to Hubert Kirchner, Miguel, How to Dress Well, Rihanna, Selena Gomez, and Descendents. Given that August 24th was a Friday, my best guess is perhaps that it was a Release Radar playlist, or perhaps an epic shuffle session.

For the top 20 bands I discovered in 2018, many of them I started listening to on Spotify, but not necessarily because of Spotify. Gilligan Moss was a discovery from a collaborative playlist shared with those that are also in a Facebook group about concert-going. I later saw them at one of the festivals I went to this year, and it even turned out that a friend knew one of the band members! Their status as my most-listened-to discovery of this year is very accurate.

Polo & Pan was a discovery from a friend, fully brought to life with a playlist built by Polo & Pan themselves and shared on Spotify. Spent some quality time sitting in a park listening to that playlist and just enjoying life. They were at the same festival as Gilligan Moss, playing the same day, making that day a standout of my concerts this year.

Karizma was a discovery from Jamie xx’s set at Outside Lands. I tracked down the song from the set with the help of several other people on the internet (not necessarily anyone I knew) and then the song that was from the set itself wasn’t even on Spotify itself (Spotify, however, did help me discover more of the artist’s back catalog, like my other favorite song ‘Nuffin Else) Apparently I was far behind the curve hearing the song from the set, since it came out in 2017 and was featured in a Chromebook ad, but Work It Out still made me lose my mind at that set. (For the record, so did Take Me Higher, a song I did not manage to track down at all, and have so much thanks for the person that messaged me on Facebook ages later to send me the link!)

Similarly, Luxxury was a DJ I first spotted on a cruise that I went on because it featured other DJs I had heard of from college, Goldroom and Gigamesh, whom I’d discovered through remixes of songs I downloaded from mp3 blogs like The Burning Ear.

~ Finding Meaning in the Platforms ~

Many of these discoveries were deepened by Spotify, or had Spotify as a vector—through a collaborative playlist, algorithmically-generated one, or the quick back-catalog access for a new artist—but don’t rely on Spotify as a platform. I prefer to keep my music listening habits platform-adjacent.

Spotify, SoundCloud, iTunes, Beatport and other music platforms I use help make my music experiences possible. But the artists making the music, performing live in venues that I have the privilege to live near and afford to visit, they are creating what keep my mind alive and energized.

The social platforms too, mediate the music-related experiences I’ve had, whether it’s with the people I share music and concert experiences with in a Facebook group, the people I exchange tracks and banter with in Slack channels, or those of you reading this on yet another platform.

I like to listen to music that moves me, physically, or that arrests my mind and takes me somewhere. More now than ever I realize that musical enjoyment for me is an intense instantiation of the continuous tension-and-release pattern that exists in so many human art forms. The waves of neatness that clash and collide in a house music track, or the soaring crescendos of harmonies.

It’s become clear to me over the years that I can’t separate my enjoyment of music from the platforms that bring me closer to it. Perhaps supporting the platforms in addition to the musical artists, performers, and venues, is just another element of contributing to a thriving music scene.

Politeness in Virtual Assistant Design

Wed, 09 May 2018 21:57:07 +0000

The wave of chatbots and virtual assistants like Cortana, Siri, and Alexa means that we’re engaging in conversations with non-humans more than ever before. Problem is, those non-human conversations can turn inhuman when it comes to social norms.

Interactions with virtual assistants aren’t totally devoid of human interaction. Indeed, they often disguise a true human interaction. Many chatbots aren’t fully automated and rely on humans to pick up the slack from the code. More fully-constructed virtual assistants like you find in Amazon’s Echo or your Apple iPhone are carefully programmed by humans. The programming choices they make also define your interactions with the personalities—and these interactions can redefine how you treat people.

A clear indication that someone is truly polite and kind is treating service people with respect, patience, and kindness. The rise of chatbots and virtual assistants, however, means that you’re never quite sure whether you’re speaking to a human. You might think that people can easily tell the difference between when they’re interacting with humans and when they’re interacting with a voice inside a smart box, but as the technology behind virtual assistants like Google Assistant, Amazon Alexa, or used by call centers evolves, that will get harder to evaluate. (Even when you’re calling a call center, it can be hard to tell whether you’ve reached a well-programmed intake bot or a real person who’s fully in the groove of their phone voice).

I find it fascinating (and saddening) that the programmers of Google Assistant’s Duplex chose to program in “umms” and “mmhmms” and did not program in any kindness indicators. Instead the voices come across as impatient and slightly condescending. I listened to the sample clips linked by Ethan Marcotte in his post Kumiho, about Google Duplex. If virtual assistants don’t include programmed kindness, the emotional labor performed by service workers will continue to be too high.

Programming to add kindness from virtual assistants is important, but so too is programming virtual assistants to expect kindness. We’re starting to be conditioned to treat chatbots as recipients for code-like commands, requiring a specific set of inputs, and those inputs do not acknowledge politeness.

It may seem overly-prescriptive, but in the same way that parents withhold items from their children until they “ask for it nicely”, it might be practical to include a “politeness mode” in virtual assistants. Hunter Walk wrote about how Amazon Alexa interactions are affecting his child, and Ben Hammersley blogged about the fact that there is no reward for politeness when he interacts with Amazon Alexa:

But there’s the rub. Alexa doesn’t acknowledge my thanks. There’s no banter, no trill of mutual appreciation, no silly little, “it is you who must be thanked” line. She just sits there sullenly, silently, ignoring my pleasantries. And this is starting to feel weird, and makes me wonder if there’s an uncanny valley for politeness. Not one based on listening comprehension, or natural language parsing, but one based on the little rituals of social interaction. If I ask a person, say, what the weather is going to be, and they answer, I thank them, and they reply back to that thanks, and we part happy. If I ask Alexa what the weather is, and thank her, she ignores my thanks. I feel, insanely but even so, snubbed. Or worse, that I’ve snubbed her.” “It’s the computing equivilent of being rude to waitresses. We shouldn’t allow it, and certainly not by lack of design. Worries about toddler screen time are nothing, compared to future worries about not inadvertently teaching your child to be rude to robots.

As virtual assistants become more common in day-to-day interactions, if they do not account for politeness, we might become a less kind society. Not only that, but impolite virtual assistants will add to the emotional labor performed by the service workers that don’t find their jobs replaced by technology.

Rediscovering Me and Moving Forward

Sat, 31 Mar 2018 10:08:33 +0000

After a breakup, how do you rediscover the activities that you enjoy and make you you?

if someone

does not want me

it is not the end of the world.

but

if i do not want me

the world is nothing but endings.

— nayyirah waheed

For myself, I spent several years in a relationship where I slowly let my own needs, wants, and desires be subsumed by those of my partner’s, and what I anticipated to be his needs, wants, and desires of me. Explicitly and implicitly, I lost myself in becoming who (I thought) he wanted me to be. After we broke up I was left with a profoundly distant sense of self. The last time I’d felt truly myself I was living at home (and that wasn’t a strong confident self). I was nothing like the person I became… or was I?

What followed has been an attempt to rediscover a sense of self and a sense of strength. I retried things I’d enjoyed with my partner in different contexts, and with different people (alone or with new friends), to derive new meaning. I needed to know if I truly enjoyed these activities or if I was only doing them because of him.

Something simple like making a bucket list helped me make real what I care about. Why would I want to go to one place instead of another? What sorts of things do I want to put on my list, activity and location-wise? How do I prioritize myself enough to get to go to those places and do those things? This also helps me tap into the sense of freedom and unpredictability in life, but in an ordered way (because that’s how I roll) that helps me discover my “true self”.

A bucket list also helped me think through shared goals, hopes, or dreams. How can I let go of a dream, or hold onto it, knowing that they might still hold that dream too? How can I travel to certain places without being reminded of them and a future I thought we’d share? How can I separate my dreams from those that we shared, created and dreamed together? Maybe I can’t. But that doesn’t mean I have to give them up. I can assess them, and see if I want to keep those goals, hopes, and dreams in my new life.

I worked to find comfort and strength in art and poetry. I asked a friend of mine for some poems about “living your best life”. I wanted some spiritual salve to learn how to remake myself after the relationship ended. She sent me poems like “My Dead Friends” by Marie Howe, “What the Living Do” by Marie Howe, and “The Journey” by Mary Oliver. I went to art museums, lingering amidst the modern art from Germany, a longtime favorite.

I also revisited things from before we started dating. I had to test things that I once cared about (to see if they still mattered to me). I’d neglected them or moved on from them or never gave myself the chance to fully commit to them. For me that was things like climbing, and going to concerts, or out dancing. I’ll attend my 100th show next month, and I recently got back from the Flash Foxy women’s climbing festival. I found myself again in the familiar experiences of going to shows, and in the community of climbing.

The crux of this process has been learning to feel like myself and like I know myself again.

Moving Forward

With a stronger sense of self, I’ve started dating again. This is hard. (All things involving people are hard). This has led me to think a lot about what makes people compatible, and what qualities are important and which ones cannot be compromised on.

I saw author Kim Culbertson speak at a panel at the Bay Area Book Festival, and she said:

“Lots of people are uncomfortable when they hold themselves up to another person and the edges don’t match.”

That’s a lot of what dating feels like (that’s a lot of what talking to other humans feels like, honestly). One inclination to ease that discomfort is to disengage—this person is different from me, so I won’t talk to them (or share much of myself with them), or befriend or date them. Another way to ease that discomfort is to soften my own edges so that the mismatched edges are less apparent. And that’s where this essay comes in.

I had to re-sharpen the edges that make me me. Now I’m working to remind myself not to soften my own edges, but instead work to find a way to appreciate mismatched edges. I don’t need to find a person that perfectly interlocks with the edges of myself to find joy, happiness, intrigue, and personal growth. I do need to find someone that appreciates my edges (and whose edges I can appreciate).

With that in mind, what does it mean to be compatible with someone? Is it the mutual appreciation of edges, or something else? I think there are various levels of it.

The surface level compatibility that provides the initial intrigue—you find each other attractive, there is some chemistry, you started talking about a shared interest.
A deeper level compatibility when you share interests or passions. It’s easy when someone shares my music taste, or shares my appreciation for music. It’s harder to appreciate the edges of someone who doesn’t like music as much as I do.
The more fundamental, deep levels of compatibility reveal themselves as you get to know someone. You start to learn whether or not your communication styles complement each other or conflict with each other. Maybe you each communicate feelings differently, or miss each other’s “love language” signals. Maybe you want to discuss deep, introspective, existential questions over lunch, and your partner just wants to eat.

The edges of the people I meet and date won’t match up with me perfectly, but part of knowing where my edges are is knowing which edges of mine need to line up with those of someone else, and which ones can be different. (I’m still learning this, and I probably always will be).

That knowledge can help me keep my edges intact as I get to know someone. There is a distinct difference between learning to appreciate or respect the interests and passions of someone else and adopting those interests and passions wholesale for myself. I’m trying them out to see what they’re like. As I experience the interests and passions of others, I might be adding new facets to my edges.

But I’m also learning that it’s okay not to share the same interests and passions as the person I’m dating. It’s enough to appreciate that they have those interests and passions. As a perfectionist, I often try to not just to be perfect, but to be perfect for someone. So I have to take a step back (often) and remind myself that other people are flawed, that I’m also flawed, and that not everyone will appreciate raisins in baked goods, or disco music, or staying up late. And that’s okay.

We’re all human, we all have edges. Keep yours sharp, and admire those of others.

This is the new year

Fri, 26 Jan 2018 22:37:27 +0000

Thinking lately

How do you decide to make a big change in life?
How do you rediscover what’s important to you?
How many concerts in a week is too many?

I’m struggling with the first one, working on the second, and am pretty sure the answer to the third one is “three”.

Reading lately

The very white ways of the top 40

The American Top 40 chart includes more dance songs, more songs performed by DJs, and significantly more white artists than its counterpart, the Billboard charts.

Shit’s racist. I used to listen to the Ryan Seacrest Top 40 driving between Chicago and Michigan because it was one of the few things that I could listen to consistently along that entire drive on just a few radio stations. It wasn’t exactly quality radio, but it kept me awake.

The Secret Lives of Playlists

The business meets somewhere at the crossroads of public relations and payola—a tradition as old as the music industry itself, historically used to define the illegal practice of record companies paying for commercial radio airtime. (Under U.S. law and FCC regulations, Payola is illegal on radio, but those laws do not apply to digital streaming platforms.) According to a 2015 Billboard article, a major-label marketing executive confirmed that pay-for-play is (or was) definitely happening.“According to a source, the price can range from $2,000 for a playlist with tens of thousands of fans to $10,000 for the more well-followed playlists.” And many are already calling the platform’s new “Sponsored Songs” endeavor a 2017 incarnation of payola.

I keep thinking I’ll get sick of Spotify thinkpieces but I’m not there yet. This one covers (in part) how Spotify structures their service to prioritize playlists over albums or other artist-created works, instead effectively reinstating payola and creating pay-to-playlists that then earn top billing all throughout the service. Me, I make my own playlists most of the time.

Can anyone turn streaming music into a real business?

Everyone wants streaming music to be cheap or free for listeners, offer every song ever recorded, be made available on every device, be consistently lucrative for the industry, and give new and established artists robust support for new music. We all want snow that isn’t cold or wet. In principle, everyone is willing to pay, and everyone is willing to compromise, but no one is willing to compromise enough.

Womp womp. This is why for all of my use and support of services like Spotify and SoundCloud, now that I can afford it, I’m trying to buy the music that matters to me when possible. Less likely to disappear that way.

Within The Context Of All Contexts: The Rewiring Of Our Relationship To Music

Old music, reframed or brought into new circulation, can be as dynamic and unpredictable as new music.

How relying on ~ the algorithms ~ has changed how we encounter music and what that means.

I Used to Insist I Didn’t Get Angry. Not Anymore.

Confronting my own aversion to anger asked me to shift from seeing it simply as an emotion to be felt, and toward understanding it as a tool to be used: part of a well-stocked arsenal.

Leslie Jamison is one of my favorite essayists, and this is no exception.

Writing Lately

I wrote two posts about analyzing my personal music data corpus. Reflecting on a decade of (quantified) music listening fits in with the rest of my blog posts about music, taking the personal tack to the quantified side of things. I also wrote up how I did all the analysis for my company blog, 10 Years of Listens: Analyzing My Music Data with Splunk. I’ve done some more analyses since these posts, like building something that lets me review the listening patterns for a specific artist compared with the dates that I’ve seen them in concert, and I’m working on analyzing if there is an average listen threshold before I see a band in concert (or not).

I also wrote about the importance that climbing has had in my life over the last year and a half in Finding Myself on the Wall. Grateful to get back on the wall tomorrow.

I took the time last year to start converting a dormant side project into a blogging series to share the links I’d collected. Calling it Borders on the Web, I post reminders of the borders that do exist on the web, as much as the techno-utopians in the world might like to pretend that they’re going away.

Listening Lately

The trend in the last year or so toward more disco vibes has been… unexpectedly awesome. Going to see at least three of these artists live in the next few months… hoping to see more music from Thunder Jackson and Disco Despair soon too.

Some great DJ sets / mixtapes on here too. Seeing the xx live last year was a highlight, almost entirely because of Jamie xx. Realized that’s a show I’d pay more than I’d like to admit to go see if it were just him DJing. Haven’t managed to see Alex Cruz yet, though he’s been in the city a couple times since I’ve been here.

Happy 2018, everyone. Feel free to follow me on Twitter if you don’t mind the occasional youtube artifact retweet.

Unexpectedly ccTLDs

Wed, 10 Jan 2018 19:44:18 +0000

Some countries have trendy ccTLDs, and startups buy in to their domain space. Vox Media has more details:

Even very small countries get ccTLDs. Here’s a close-up of the area around Australia and the many small island nations that have their own domain names. Some of these countries realized that they could make a lot of money if they opened their domains to foreigners. The result: popular websites like last.fm (.fm is the domain of the Federated States of Micronesia) and twitch.tv (.tv is the domain for the island nation of Tuvalu). The .io domain, assigned to the British Indian Ocean Territory, has become popular among programmers. They associate the domain with the technical term input/output and use it to create “artisinal websites.”

Best of 2017: Live Shows

Sun, 31 Dec 2017 20:39:24 +0000

My favorite shows of 2017. Here’s to more great ones in 2018!

October 27, 2017: DJ Aaron Axelson, Lewis Ofman, Yelle

Rickshaw Stop, San Francisco CA Popscene became my favorite concert sponsor this year, in no large part because of the skills of their DJs. This show surpassed my low expectations to be a great time of dancing and grooving and new music discoveries.

February 23, 2017: Rad Dad, Gibbz

The Hotel Utah Saloon, San Francisco CA A local band opened for an undersung nu-disco artist, Gibbz. A great way to open p Noise Pop week 2017, and unexpectedly great sound quality for such a small space. Excited to see Gibbz play again next year.

September 19, 2017: NVDES, RAC

The Independent, San Francisco CA RAC has put on a spectactularly dance-able show every time I’ve seen them. This most recent adventure did not disappoint.

April 16, 2017: Sampha, The XX

Bill Graham Civic Auditorium, San Francisco CA I would pay Jamie XX to DJ my life, but I can’t afford it. I could afford this show, though. It was incredible. Sampha was great too. Highlight: a mirror that appeared partway through the set that gave the audience a view of Jamie XX’s DJing and his dorky dance moves.

September 13, 2017: The Dirty Nil, Bleached, Against Me!

Regency Ballroom, San Francisco CA Just as good as they were 10 years ago when I saw them in Chicago, if not better. A restorative and energetic show.

February 4 2017: Wheatus, Mike Doughty

The Independent, San Francisco CA Wheatus played old hits and new jams, and Mike Doughty pulled them out to back him as he played a bunch of Soul Coughing songs. I was there more for his solo songs, but the artistry and adventure of his live conducting of the band behind him made for an incredible show that was supremely groove-able.

Best of 2017: Books

Sun, 31 Dec 2017 20:15:02 +0000

The best books I read this year, loosely categorized.

Favorite Book

Uproot: Travels in 21st-Century Music and Digital Culture

Fantastic. Jace Clayton has an unnervingly well-placed finger on the pulse of modern music culture, in a way that makes you feel out-of-touch no matter how much music you listen to. I feel like I understand the music industry, global commerce, music-making, and people around the world better after reading this book. It blends together all those aspects and manages to be writing about music without making you miss the music (but the website for the book has playlists, just in case you do). A personal non-fiction book, a style I turn out to like quite a bit (Word by Word has a similar style).

Beyond Historical Fiction

The Atlas of Forgotten Places

A book picked up at the library on a whim turned out to be one of my favorite books of the year. Tying history with stories of personal struggle and tragedy, this doesn’t tie up neatly and doesn’t come across as try-hard either. A reminder of reality in a novel.

The Nightingale

This story follows two sisters in World War II through their wartime decisions and the present-day. Not quite as brilliant as “All The Light We Cannot See” but just as moving.

Manhattan Beach

Egan’s research shows in the vividness of the storytelling and the mental imagery constructed. You can feel the weight of the decisions made by the characters and their physical burdens in the novel.

The Three-Body Problem

I didn’t manage to finish the trilogy, but this novel stunningly takes the prospect of alien contact and puts it in context of Communist China, with some perspective from competing American, and Russian global interests too. Reading it the same year as Arrival (Stories of Your Life and Others) leads to echoes of similar themes, but the approach is so vastly different that I only thought of the comparison in writing this, not in reading the novel. This book is solidly sci-fi, but the role of history seemed so relevant to the story that I’m categorizing it here.

The Japanese Lover

The first I’ve read by Isabel Allende, and a love story hidden inside a story about the Japanese internment during World War II and the havoc it wreaked on families, alongside the present-day immigrant experience in the United States.

Pleasant Feel-Good Discoveries

The Hating Game

A brand new author on the romance novel scene wrote this and it is delightful. Doesn’t rely over-much on existing romance novel tropes, and manages to be well-written even while you’re rolling your eyes occasionally. Depicts the internal struggle that prevents many of us from believing something is real all too well.

The Royal We

This may be loosely Kate and William fanfic, but I. Am. Here. For. This. It made the rounds at book swap this year and I maintain that it took the classic trope of “ordinary person meets royal but doesn’t know they’re royal” and makes it unexpected and a delight.

Morgan Matson novels

The Sarah Dessen novels of a new age, with less tragic character backstories. Enjoyable discoveries for this year, and I’m looking forward to her next one due out next year. Eligible Curtis Sittenfeld is a delight. I didn’t realize this was a Pride and Prejudice rewrite until the end, and that made me like it more. An enjoyable read that helps you realize just how much of modern romance fiction is based on the tropes (first?) established in Pride and Prejudice.

Proper Literature or Vague Classics

The Unwomanly Face of War

This was devastating. A vivid window into the reality and the legacy of women who fought for or worked for the Soviet Union in World War II, her work manages to be both a record of history and an critical eye cast toward the Soviet government. Just as unrelenting as Voices of Chernobyl. I am inclined to seek out all of her work.

Stories of Your Life and Others

The first eight stories were great. Skip the rest of the collection. Perfect for the overly-analytical people that try to analyze rather than experience their emotions. The film Arrival was based on one of these stories.

On Immunity

A beautiful book of personal essays interwoven with research. Brings the human back to science and medicine. Also swapping this book at book swap led to my first encounter with “the first page” and my friends’ desire to have me read the first page of books aloud for a podcast.

Snow Crash

Finally read this novel and it has stayed vividly with me over the past few months since reading it. A clear precursor to so many novels that followed it, and a great reminder that what is online is never truly only online.

Fantasy

Graceling Realm series

Court of Thorns and Roses series

Six of Crows duology

I grouped these three series together because they handled in varying degrees:

Mind control and/or a race of superpowered/magical people
Romance (from hints at beginnings of love, to explicit seduction)
Warring states and the steps that those embroiled among them must take to win power
Redemption of the self in the face of personal insecurities

The Graceling series was the best of these three, I’d wager. I read the third and the second books in the wrong order on accident, and might prefer that order instead of the intended order. That could be because I’m a less attentive reader than some. For a focus on heist and revenge adventures, read the Six of Crows duology. Not much of a romance thread through these books, it focuses more on coming of age and learning what matters. For the most romance, make it through the near-insufferable first book of the Court of Thorns and Roses series and follow it through to the end of the third book (then reconsider rereading the first book). The next few books that aren’t out yet are spinoffs, so if you, like me, have a rule about not starting series before they end, never fear. This series has the most similar tropes to the Graceling series, so consider reading them far apart.

Best of 2017: Newly-Discovered Music

Sun, 31 Dec 2017 16:39:24 +0000

I used my music data to look up my favorite artists that I discovered in 2017. These are the ones that are the memorable favorites, beyond the statistical favorites.

Pional

This one is a surprise but a good reminder that small obsessions can make a big difference in overall statistics. I have The Burning Ear to thank for this discovery, and Spotify for entertaining it. Song recommendation: Casualty.

R.Lum.R

I discovered this artist because they’re touring as the headliner with Gibbz, who I was already familiar with. The groovy vibe of this artist took those tickets from a probable insta-purchase to an actual insta-purchase. Song recommendation: Love Less.

Jason Gaffner

A discovery thanks to The Burning Ear, I discovered Jason Gaffner’s nu-disco grooves around the same time that I got obsessed with some songs by Gibbz (who I must’ve discovered in 2016). I bought this song soon after and am keeping an eye out for new releases. Song recommendation: Murder in the First Degree.

Alex Cruz

I heard Alex Cruz for the first time when I was in Greece, listening to a set that my friend started playing. It took me three tries to figure out who she was talking about, and then I discovered a few of his sets that he puts out as the Deep and Sexy Podcast. Song recommendation: Deep Sexy Podcast 29 Alive Afrikaburn.

Perfume Genius

I can’t remember if I started listening to Perfume Genius because of Discover Weekly or the Song Exploder podcast, but damn they’re good. My only regret is that I discovered them too late to get tickets to their sold out show. Song recommendation: Song Exploder episode.

Super Duper

I don’t remember how I discovered this artist. I think it was an autoplay on SoundCloud after listening to some tracks The Burning Ear had posted? Either way, I fell in love with this remix. Song recommendation: Bon Iver - Over Soon Super Duper Remix.

Shallou

I came across this band on The Burning Ear too. I think they’ll be around for Noise Pop next year so I’ll have to decide if I want to go see them. I’m mostly in love with this song. Song recommendation: Slow.

Sampha

He opened for the xx, so I checked out his Spotify page after I found out he was opening for them. Sweet, sweet grooves. Song recommendation on YouTube.

James Barrett

This guy showed up in my Discover Weekly playlist. I really like this song, but didn’t get as into the rest of his songs. Still a damn good song tho. Song recommendation: College.

Ella Vos

I enjoyed her song Little Brother so much that I got tickets to see her next year. I’ll be keeping an eye out for new releases from her as well. Song recommendation: Little Brother.

Less notable discoveries

Jane

I came across this band on SoundCloud through The Burning Ear again. This song was an easy purchase because it’s so catchy. Song recommendation: We Don’t Wanna Dance.

Bjéar

This artist showed up on my Discover Weekly playlist. Great for fans of Bon Iver. Song recommendation: Going to the Sun.

Imad Royal

This was another The Burning Ear discovery, and an easy purchase! Song recommendation:

The Full List:

The full list of 35 artists that had more than 10 listens each, first listened to in 2017:

Artist	Listens	Tracks
Pional	42	A New Dawn As Time Was Passing By Casualty In Another Room Invisible / Amenaza It’s All Over It’s All Over - John Talabot’s Stripped Refix Of My Mind The Way That You Like
Alex Vargas	41	7 Sins Ashes Follow You Giving Up The Ghost Higher Love Inclosure Indivisible Oh Love, How You Break Me Up Renegade Shackled Up Solid Ground Sweet Abandon Warnings Wear Your Demons Out
Jason Gaffner	34	Feel Something Feel Something (Garruda Remix) Losing My Mind Losing My Mind (3 Monkeyzz Remix) Murder In The First Degree Murder In The First Degree (Aristo G Remix) Phantom Phantom (Keljet Remix) When The Sun Goes Down
Sampha	30	(No One Knows Me) Like The Piano Beneath The Tree Blood On Me Happens Incomplete Kisses Kora Sings Plastic 100°C Reverse Faults Take Me Inside Timmy’s Prayer Too Much Under What Shouldn’t I Be?
Kyko	28	Animals Dive In Drive Headlights Hideaway Horizon Mexico Native Nature Pull Me Up
R.Lum.R	25	Be Honest Be Honest (Attom Remix) Bleed Into The Water Close Enough Frustrated Frustrated - Russ Macklin Remix Learn Love Less Nothing New Show Me Suddenly Tell Me
Utah	25	02:12 Hail the Underdog In Slow Motion Lights Out Mirrors No Coast On the Mountain by the Sea One Million People of the Future SFSG Still Good Watercolor When People Come Together
Young Summer	25	Alright Alright (Karl Kling Remix) Blood Love Echo Fallout Old Chunk of Coal Sons Of Lightning (Super Duper Remix) Taken Waves That Rolled You Under (backstroke. Remix)
Ralph	23	Busy Man Cold to the Touch Cold to the Touch - Nicolaas Remix Screenplay Something More Tease This Is Funky
Alex Cruz	21	Haunting - Original Mix Haunting - Radio Edit Haunting - Sebastien Radio Edit Haunting - Sebastien Remix Haunting [ANR063] - Sebastien Remix Rubberband - Radio Edit Shoreline - Extended Mix Sweet Child Sweet Child - Club Mix Sweet Child - Extended Sweet Child - Original Mix
National Parks	21	Backwards Centaur Five Hour Winnipeg Julia Long Winter The Plural of Moose Is Moose
Bien	20	Confetti Crowd Goes Wild Electric Dream Flashback Last Man Standing Must Be Dreaming Spinning on Blue Stars Across the Sky The Best Part
Perfume Genius	20	Body’s In Trouble - Recorded at Spotify Studios NYC Choir Die 4 You Every Night Go Ahead Just Like Love Otherside Sides Slip Away Slip Away - Recorded at Spotify Studios NYC Valley Wreath Wreath (Kaitlyn Aurelia Smith Remix)
Super Duper	20	Angela Angela [Thissongissick.com Premiere] Don’t Worry Finale (feat. Ruelle) Finale Ft. Ruelle Hollow (feat. Quinn Lewis) Innocence (feat. REMMI) Innocence (feat. REMMI) (LUCA LUSH Remix) [NEST HQ Premiere] Innocence (feat. REMMI) (Madeaux Remix) [NEST HQ Premiere] Innocence Ft. Remmi Makes The Wind Ft. Remmi & Jung Youth Makes the Wind (feat. REMMI & Yung Youth) Never Gets Old (feat. Remmi) Revival Second Chances (feat. Louis Johnson) Undercover Ft. Patrick Baker
Emerson Jay	18	Fake It Slow Feel Like Gold LZY Me Light Out Move Perspective Secret City Smok Take Take Take Tru War When It’s Night
Ruby Empress	17	Danseuse De Delphes Deluca Escapism Deluxe Kimono House Lovelight (JV-30) Strung Out The Empress
Ella Vos	16	00000 Million - Recorded at Spotify Studios NYC Little Brother White Noise
Majik	16	27 Closer High How It Is It’s Alright Paralysed Real - Skeleton Mix Save Me Talk to Me
à la mer	16	Abroad ~ Say That You Want It Abroad ~ Time
Imad Royal	15	Bad 4 U Bad 4 U - Light House Remix Down For Whatever (feat. Pell) Losing It All Smile Troubles
Mr Sanka	15	Be Easy Flight Mode Flight Mode (Jengi Beats Remix) Flight Mode (Lauer Remix) Forever and a Day Gallon Gallon (Cassian Remix) Midnight Air Midnight Air (JAQ Remix) Midnight Air - JAQ Remix
Crooked Colours	14	Another Way Capricious (Benson Remix) Capricious (Paces Remix) Come Down Come Down [Alison Wonderland Remix] Flow Flow - Extended Re-Rub In Your Bones In Your Bones (Chiefs Remix) Step
Rex Orange County	14	A Song About Being Sad BEST FRIEND Corduroy Dreams Edition Green Eyes, Pt. II Loving Is Easy Paradise Uno
Shallou	14	. . . Love Begin (feat. Wales) Begin - Recorded at Spotify Studios NYC Fictions Friends - Recorded at Spotify Studios NYC Heights Heights - Extended Mix Motion Picture Soundtrack Slow You and Me
James Barrett	13	College Marrow Rodger The Metamorphosis You Used to Remind Me of the Sky
Klyne	13	Break Away (FaltyDL Remix) Closer Don’t Stop Don’t Stop - Boston Bun Remix Entropy Lend Me Another Name Sure Thing - Lxury Remix Waiting Wit U
Liv Dawson	13	Hush Last Time - Live At RAK Open Your Eyes Painkiller Painkiller - Acoustic Reflection Searching Still Tapestry
bjéar	13	Big Sky Cold Firefall Firefall - Radio Edit Going to the Sun Hymn Nell Nevada Tuolumne
Jane	12	Sister We Don’t Wanna Dance
Sean McVerry	12	Kerosene Marcy and the Apparition Motion Picture Films Natalie Strangers Tiger Lily
Charles Fauna	11	Abandon Hypnosis Hypnosis - Brothertiger Remix Liaison Myth Restless Child
Ed Tullett	11	Faux In Cure Kadabre Malignant Posturer Silver Dive
Maggie Rogers	11	Alaska Alaska - Sofi Tukker Remix Alaska - Toby Green Remix Dog Years On + Off
Polish Club	11	Able Beeping Did Somebody Tell Me Don’t Fuck Me Over My House
Shy Girls	11	Arrest Me (Noah Breakfast Remix) [feat. Tei Shi] Out of Touch (feat. Rome Fortune) Say You Will Time After Time Trivial Motion Watercolor Dreams Why I Love

Reflecting on a decade of (quantified) music listening

Tue, 19 Dec 2017 19:25:30 +0000

I recently crossed the 10 year mark of using Last.fm to track what I listen to.

From the first tape I owned (Train’s Drops of Jupiter) to the first CD (Cat Stevens Classics) to the first album I discovered by roaming the stacks at the public library (The Most Serene Republic Underwater Cinematographer) to the college radio station that shaped my adolescent music taste (WONC) to the college radio station that shaped my college experience (WESN), to the shift from tapes, to CDs, (and a radio walkman all the while), to the radio in my car, to SoundCloud and MP3 music blogs, to Grooveshark and later Spotify, with Windows Media Player and later an iTunes music library keeping me company throughout…. It’s been quite a journey.

Some, but not all, of that journey has been captured while using the service Last.fm for the last 10 years. Last.fm “scrobbles” what you listen to as you listen to it, keeping a record of your listening habits and behaviors. I decided to add all this data to Splunk, along with my iTunes library and a list of concerts I’ve attended over the years, to quantify my music listening, acquisition, and attendance habits. Let’s go.

What am I doing?

Before I get any data in, I have to know what questions I’m trying to answer, otherwise I won’t get the right data into Splunk (my data analysis system of choice, because I work there). Even if I get the right data into Splunk, I have to make sure that the right fields are there to do the analysis that I wanted. This helped me prioritize certain scripts over others to retrieve and clean my data (because I can’t code well enough to write my own).

I also made a list of the questions that I wanted to answer with my data, and coded the questions according to the types of data that I would need to answer the questions. Things like:

What percentage of the songs in iTunes have I listened to?
What is my artist distribution over time? Do I listen to more artists now? Different ones overall?
What is my listen count over time?
What genres are my favorite?
How have my top 10 artists shifted year over year?
How do my listening habits shift around a concert? Do I listen to that artist more, or not at all?
What songs did I listen to a lot a few years ago, but not since?
What personal one hit wonders do I have, where I listen to one song by an artist way more than any other of their songs?
What songs do I listen to that are in Spotify but not in iTunes (that I should buy, perhaps)?
How many listens does each service have? Do I have a service bias?
How many songs are in multiple services, implying that I’ve probably bought them?
What’s the lag between the date a song or album was released and my first listen?
What geographic locations are my favorite artists from?

As the list goes on, the questions get more complex and require an increasing number of data sources. So I prioritized what was simplest to start, and started getting data in.

Getting data in…

I knew I wanted as much music data as I could get into the system. However, SoundCloud isn’t providing developer API keys at the moment, and Spotify requires authentication, which is a little bit beyond my skills at the moment. MusicBrainz also has a lot of great data, but has intense rate-limiting so I knew I’d want a strategy to approach that metadata-gathering data source. I was left with three initial data sources: my iTunes library, my own list of concerts I’ve gone to, and my Last.fm account data.

Last.fm provides an endpoint that allows you to get the recent tracks played by a user, which was exactly what I wanted to analyze. I started by building an add-on for Last.fm with the Splunk Add-on Builder to call this REST endpoint. It was hard. When I first tried to do this a year and a half ago, the add-on builder didn’t yet support checkpointing, so I could only pull in data if I was actively listening and Splunk was on. Because I had installed Splunk on a laptop rather than a server in ~ the cloud ~, I was pretty limited in the data I could pull in. I pretty much abandoned the process until checkpointing was supported.

After the add-on builder started supporting checkpointing, I set it up again, but ran into issues. Everything from forgetting to specify the from date in my REST call to JSON path decision-making that meant I was limited in the number of results I could pull back at a time. I deleted the data from the add-on sourcetype many times, triple-checking the results each time before continuing.

I used a python script (thanks Reddit) to pull my historical data from Last.fm to add to Splunk, and to fill the gap between this initial backfill and the time it took me to get the add-on working, I used an NPM module. When you don’t know how to code, you’re at the mercy of the tools other people have developed. Adding the backfill data to Splunk also meant I had to adjust the max_days_ago default in props.conf, because Splunk doesn’t necessarily expect data from 10+ years ago by default. 2 scripts in 2 languages and 1 add-on builder later, I had a working solution and my Last.fm data in Splunk.

To get the iTunes data in, I used an iTunes to CSV script on Github (thanks StackExchange) to convert the library.xml file into CSV. This worked great, but again, it was in a language I don’t know (Ruby) and so I was at the mercy of a kind developer posting scripts on Github again. I was limited to whatever fields their script supported. This again only did backfill.

I’m still trying to sort out the regex and determine if it’s possible to parse the iTunes Library.xml file in its entirety and add it to Splunk without too much of a headache, and/or get it set up so that I can ad-hoc add new songs added to the library to Splunk without converting the entries some other way. Work in progress, but I’m pretty close to getting that working thanks to help from some regex gurus in the Splunk community.

For the concert data, I added the data I had into the Lookup File Editor app and was up and running. Because of some column header choices I made for how to organize my data, and the fact that I chose to maintain a lookup rather than add the information as events, I was up for some more adventures in search, but this data format made it easy to add new concerts as I attend them.

Answer these questions…with data!

I built a lot of dashboard panels. I wanted to answer the questions I mentioned earlier, along with some others. I was spurred on by my brother recommending a song to me to listen to. I was pretty sure I’d heard the song before, and decided to use data to verify it.

I’d first heard the song he recommended to me, Waiting on the Summer, in March. Hipster credibility: intact. Having this dashboard panel now lets me answer the questions “when was the first time I listened to an artist, and which songs did I hear first?”. I added a second panel later, to compare the earliest listens with the play counts of songs by the artist. Maybe the first song I’d heard by an artist was the most listened song, but often not.

Another question I wanted to answer was “how many concerts have I been to, and what’s the distribution in my concert attendance?”

It’s pretty fun to look at this chart. I went to a few concerts while I was in high school, but never more than one a month and rarely more than a few per year. The pace picked up while I was in college, especially while I was dating someone that liked going to concerts. A slowdown as I studied abroad and finished college, then it picks up for a year as I get settled in a new town. But after I get settled in a long-term relationship, my concert attendance drops off, to where I’m going to fewer shows than I did in high school. As soon as I’m single again, that shifts dramatically and now I’m going to 1 or more show a month. The personal stories and patterns revealed by the data are the fun part for me.

I answered some more questions, especially those that could be answered by fun graphs, such as what states have my concentrated music listens?

It’s easy to tell where I’ve spent most of my life living so far, but again the personal details tell a bigger story. I spent more time in Michigan than I have lived in California so far, but I’ve spent more time single in California so far, thus attending more concerts.

Speaking of California, I also wanted to see what my most-listened-to songs were since moving to California. I used a trellis visualization to split the songs by artist, allowing me to identify artists that were more popular with me than others.

I really liked the CHVRCHES album Every Open Eye, so I have three songs from that album. I also spent some time with a four song playlist featuring Adele’s song Send My Love (To Your New Lover), Ariana Grande’s Into You, Carly Rae Jepsen’s Run Away With Me, and Ingrid Michaelson’s song Hell No. Somehow two breakup songs and two love songs were the perfect juxtaposition for a great playlist. I liked it enough to where all four songs are in this list (though only half of it is visible in this screenshot). That’s another secret behind the data.

I also wanted to do some more analytics on my concert data, and decided to figure out what my favorite venues were. I had some guesses, but wanted to see what the data said.

The Metro is my favorite venue in Chicago, so it’s no surprise that it came in first in the rankings (I also later corrected the data to make it its proper name, “Metro” so that I could drill down from the panel to a Google Maps search for the venue). First Midwest Bank Ampitheatre hosted Warped Tour, which I attended (apparently) 5 times over the years. Since moving to California it seems like I don’t have a favorite venue based on visits alone, but it’s really The Independent, followed by Bill Graham Civic Auditorium, which doesn’t even make this list. Number of visits doesn’t automatically equate to favorite.

But what does it MEAN?

I could do data analysis like that all day. But what else do I learn by just looking at the data itself?

I can tell that Last.fm didn’t handle the shift to mobile and portable devices very well. It thrives when all of your listening happens on your laptop, and it can grab the scrobbles from your iPod or other device when you plug it into your computer. But as soon as internet-connected devices got popular (and I started using them), listens scrobbled overall dropped. In addition to devices, the rise of streaming music on sites like Grooveshark and SoundCloud to replace the shift from MediaFire-hosted and MegaUpload-hosted free music shared on music blogs also meant trouble for my data integrity. Last.fm didn’t handle listens on the web then, and only handles them through a fragile extension now.

Distinct songs and artists listened to in Last.fm data.But that’s not the whole story. I also got a job and started working in an environment where I couldn’t listen to music at work, so wasn’t listening to music there, and also wasn’t listening to music at home much either due to other circumstances. Given that the count plummets to near-zero, it’s possible there were also data issues at play. It’s imperfect, but still fascinating.

What else did I learn?

I have a lot of songs in my iTunes library. I haven’t listened to nearly 30% of them. I’ve listened to nearly 25% of them only once. That’s the majority of my music library. If I split that by rating, however, it would get a lot more interesting. Soon.

You can’t see the fallout from my own personal Music-ocalypse in this data, because the Library.xml file doesn’t know which songs don’t point to actual files, or at least my version of it doesn’t. I’ll need more high-fidelity data to determine the “actual” size of my library, and perform more analyses.

I need more data in general, and more patience, to perform the analyses to answer the more complex questions I want to answer, like my listening habits of particular artists around a concert. As it is, this is a really exciting start.

If you want more details about the actual Splunking I did to do these analyses, I’ll be posting a blog on the official Splunk blog. That got posted on January 4th! Here it is: 10 Years of Listens: Analyzing My Music Data with Splunk.

Yoga Beta for Climbers

Mon, 06 Nov 2017 17:02:30 +0000

As a companion to Finding Yourself on the Wall, sometimes what you need while climbing isn’t real beta or advice of what to do, but mental reinforcement. This beta can sound kind of like the mantras that someone might give you in the midst of a yoga class—yoga beta.

Do what feels right
Don’t forget to breathe
Don’t look, just feel
You are stronger than you think
Just let go

Finding Myself on the Wall

Mon, 06 Nov 2017 17:00:13 +0000

How climbing teaches me to manage my fear and love myself.

Sometimes I find myself on the wall doing something I never thought possible: holding onto something that doesn’t seem to have a place to hold, or reaching something that looks out of reach. Other times it’s like I’m waking up to find myself trapped in what seems to be an inescapable spot: no holds above me, or nowhere to put my feet to push myself higher. In these cases, the problem is clear. The solution isn’t.

In climbing, the problem can be on the wall, or it can be with my confidence, or my fear. Being able to consistently test solutions, push through challenges, and conquer the problem is what makes climbing a perfect mental and physical outlet for me.

For me, climbing is all about managing fear and trusting myself. I have to manage my natural instincts of being afraid of heights and of falling. I also have to learn to trust my abilities and skills while respecting myself and my boundaries in order to avoid getting hurt or endangering myself or others.

In addition, the different types of climbing require different levels of this fear management and self-trust. I first learned top-rope climbing, but as I got better I got more comfortable. Then I learned bouldering, and got more comfortable there, so I learned how to lead climb. Throughout this process, I’ve built my physical strength and climbing technique, but also self confidence and my ability to manage fear.

Top-roping is the most comfortable form of climbing for me. I can see the rope, and I can sometimes see the anchor keeping the rope secure. I can feel the taut lack of slack in my rope, and lean back from the wall to test it. I can rest at any time as well, so there is time to slow down and take breaks. All of this physical security reinforces a psychological sense of security, which can help me do more challenging moves and climb higher than I might otherwise feel comfortable climbing.
Bouldering requires me to stomach my fear and muster my self-confidence to take me to the top of a wall, or over the top of a wall, without a rope. Bouldering routes are typically anywhere from 10-20 feet high in a gym, and in some incredible outdoor routes, 40 or more feet high. Without the physical security of a rope or an anchor, I have to know my physical and psychological strengths and limits before I start. This forces me to scope out the route before I start climbing, and prepare myself to jump or fall to the ground if I feel uncomfortable. Bouldering forces me to get used to this discomfort and either overcome it or recognize when it is valid and to listen to it.
Lead climbing takes the height of top-roping and combines it with the mental aspects of bouldering. No longer do I have the visible anchor or a taut rope to help me feel safe—it’s just me and the wall. I’m conquering the problem while also taking all the necessary steps to keep myself safe: clip properly, climb safely around the rope, and rest when I can. There’s little to no room for fear.

Each type of climbing removes an element of physical security and further challenges my psychological security as I progress. In this way, I’ve been forced to progressively confront and challenge my limits at the same time that I learn to respect and recognize them.

The dangers of climbing are real. It’s an extreme sport. Though it doesn’t always feel dangerous in a gym, any time that you are high up in the air relying on humans and equipment, something can fail and you can die. It’s also easy to get injured due to bad technique: over-gripping holds, inadequately engaging muscles, straining hand muscles and tendons on hard-to-grip holds. If anything, these risks force me to prioritize muscle recovery and rest days, allowing me to recognize that just as physical self care is important, so too is psychological self care.

Despite these risks, climbing lets me get more in-tune with myself than anything else that I’ve tried. It’s a wall of problems, but each one is recognizable and each one is solvable, and I can try them again and again. I can learn by watching someone else solve it, but I can’t solve it the same way because we have different skill sets, physical strength, and body types. I still have solve the problem myself in my own way.

Climbing with other people has also been key to my mental strength. Climbing partners are vital to my safety, but also to my confidence level. They can encourage me to try new routes, and give me beta when I start to falter on a route. Beta, typically defined as information about a route, can also involve encouragement. Everything from the tactical “there’s a foothold by your right knee” to the encouraging “you can reach it!” to the calming “don’t look, just feel” is great beta that has helped me succeed. (I’ve named that last type Yoga Beta). Even so, sometimes the best beta is silence so that I can focus on the problem.

Climbing as a method for teaching myself that I can succeed and iterating my way through problem-solving helps me overcome my fear of failure. I’m learning to trust myself to get through each move, and find something to (physically, psychologically) support myself along the way. I have to trust myself, and the rock, every step of the way.

Country-specific search results

Tue, 31 Oct 2017 17:36:52 +0000

It isn’t really possible to search the “global web” today. You can, however, try to use Google to search the web of another country by manually manipulating the ccTLD in the URL to divert your search to a different country service than the country you are located in.

But starting recently, that’s no longer possible. Betanews points out that Google makes it harder to search for results from other countries:

Google has announced that it will now always serve up results that are relevant to the country that you’re in, regardless of the country code top level domain names (ccTLD) you use.

What can you do instead? The official Google blog explains in Making search results more local and relevant:

If for some reason you don’t see the right country when you’re browsing, you can still go into settings and select the correct country service you want to receive. Typing the relevant ccTLD in your browser will no longer bring you to the various country services—this preference should be managed directly in settings.

This codifies your country preference, making it harder to switch across different experiences. In the past I’ve both searched in different languages and modified the ccTLD to attempt to locate different search results. Now my searches are limited to the information stored in the US-specific country service maintained by Google, unless I make a settings-level change to affect that.

Searching the global web for information gets a little bit harder. Perhaps market research is showing Google that our hunt for information is more valuable when it’s local (or more “relevant” at least). It’s another way that the web is mediated for our consumption.

Who gives you the Internet?

Fri, 27 Oct 2017 21:22:36 +0000

Iran and Russia are becoming Internet provider nexuses to other countries. Dyn Research wrote about shifts in 2013 that led states in the Persian Gulf to seek out additional Internet providers.

Sometimes, it takes a real disaster to create something genuinely new. March 2013 was a month of disasters in the Middle Eastern, South Asian, and East African Internet, with major submarine cable cuts affecting SMW3, SMW4, IMEWE, EIG, SEACOM, and TE-North.

One of the “genuinely new” Internet traffic paths that emerged in response is a counterintuitive terrestrial route, linking the ancient Indian Ocean trade empire of Oman with the Internet markets of Western Europe, by way of Iran, Azerbaijan, and the Russian Caucasus. As we’ll see, its effects are now being felt across the region, from Pakistan, to Gulf states like Bahrain and Oman, to Kenya.

More recently, Russia started providing a new Internet link to North Korea. As reported by 38North, Russia Provides New Internet Connection to North Korea:

The connection, from TransTeleCom, began appearing in Internet routing databases at 09:08 UTC on Sunday, or around 17:38 Pyongyang time on Sunday evening.

Before this additional route became available, China was the only provider of Internet access to the country’s sole ISP.

Until now, Internet users in North Korea and those outside accessing North Korean websites were all funneled along the same route connecting North Korean ISP Star JV and the global Internet: A China Unicom link that has been in operation since 2010.

This additional link makes the country’s access to the Internet less precarious and vulnerable to disconnection by attackers.

More than once the link has been the target of denial of service attacks. Most were claimed by the “Anonymous” hacking collective, but on at least one previous occasion, many wondered if US intelligence services had carried out the action.

IPv4 trafficking in Romania

Thu, 26 Oct 2017 18:09:44 +0000

Romania is selling IPv4 addresses to make money, but how did they get so many in the first place? ComputerWorld explores How Romania’s patchwork Internet helped spawn an IP address industry.

The roots of the Romanian IP address trade lie in the country’s peculiar Internet history. When commercial Internet service began in Romania around 2000, it was totally unplanned and unregulated. People started ISPs by pulling cables from one house to the next.

As the IPv6 transition and adoption is still ongoing, people are still seeking out IPv4 addresses where they can find them.

Where is the Internet decentralized?

Wed, 25 Oct 2017 18:06:51 +0000

Dyn Research interrogates the notion that the Internet is decentralized by looking at the actual state of infrastructure and routing resilience around the world. What did they find?

The key to the Internet’s survival is the Internet’s decentralization — and it’s not uniform across the world. In some countries, international access to data and telecommunications services is heavily regulated. There may be only one or two companies who hold official licenses to carry voice and Internet traffic to and from the outside world, and they are required by law to mediate access for everyone else.

Countries might not be ensuring maximum redundancy through decentralization for political reasons, a desire to control access to the Internet more easily, but also due to monetary reasons:

Increased diversity at the international frontier often spells less money for the national incumbent provider (typically the old telephone company, often owned by the government itself). Without some strong legal prodding and guidance from the telecoms regulator, significant diversification in smaller markets with a strong incumbent can take a long, long time.

Unrepresented languages on the web

Tue, 24 Oct 2017 18:00:22 +0000

Per Al Jazeera, 95% of the world’s languages continue to be unrepresented online.

The real problem is a digital architecture that forces people to operate on the terms of another culture, unable to continue the development of their own.

The architecture of the web influences the languages and cultures interacting with it:

He rightly homes in on the invisible underpinnings that enable us to use a language online, such as input methods, OS support (on a range of devices, in countless applications), transliteration and translation and spell-checking tools. Just developing a Yiddish spell-checker, for instance, has required a stable input method for the modified Hebrew alphabet that Yiddish uses, the prior standardization of that alphabet (still contested), standardized spellings of most words (sometimes contested), technical ease in handling the Yiddish alphabet and a loaded dictionary.

It’s complex to reflect the world views and cultures of the world on the web.

Each language reflects a unique world-view and culture complex, mirroring the manner in which a speech community has resolved its problems in dealing with the world, and has formulated its thinking, its system of philosophy and understanding of the world around it. In this, each language is the means of expression of the intangible cultural heritage of people, and it remains a reflection of this culture for some time even after the culture which underlies it decays and crumbles, often under the impact of an intrusive, powerful, usually metropolitan, different culture.

What makes it such a challenge to both incorporate multiple languages on the web, and also to build out fleshed-out versions of those languages?

Language homogenization on the web

Mon, 23 Oct 2017 17:59:47 +0000

Motherboard says The Internet Is Killing Most Languages:

The great flat, globalized world of the internet operates pretty much as a monoculture, Kornai says. Only about 250 languages can be called well-established online, and another 140 are borderline. Of the 7,000 languages still alive, perhaps 2,500 will survive, in the classical sense, for another century, and many fewer will make it on to the internet.

Globalization of the world and the web could lead to homogenization of the languages in both places.

The adage “If it’s not on the web, it does not exist,” neatly encapsulates the loss of prestige. And as a generation of digital natives comes up, their online tongue is likely not to be their mother tongue—a loss of competence.

Languages on the web matter for self identity

Fri, 20 Oct 2017 17:58:48 +0000

Is homogenization of language on the web an instantiation of totalitarianism? Boston Review on Herta Müller’s Language of Resistance:

Since language plays such an important part in the construction of the self, when the state subjects you to constant acts of linguistic aggression, whether you realize it or not, your sense of who you are and of your place in the world are seriously affected. Your language is not just something you use, but an essential part of what you are. For this reason any political disruption of the way language is normally used can in the long run cripple you mentally, socially, and existentially. When you are unable to think clearly you cannot act coherently. Such an outcome is precisely what a totalitarian system wants: a population perpetually caught in a state of civic paralysis.

What if the web is the state, in this context? What does it mean for self-identity, power, and a neutral web?

Top-level domains and nationalism

Thu, 19 Oct 2017 22:50:24 +0000

In 2010, Irina Shklovski and David M. Struthers wrote an excellent article on Kazakh national identity and its reflection through top-level domain name choices. The article is titled: Of States and Borders on the Internet: The Role of Domain Name Extensions in Expressions of Nationalism Online in Kazakhstan and the Oxford Internet Institute makes a PDF available.

The space on the internet is easily traversable and state boundaries in the form of domain extensions can be crossed with no more effort than a click of a mouse. Yet, what might such traversals of imagined state boundaries on the internet mean to the people doing the traversing? This question is especially relevant when considering people from Kazakhstan, a country where notions of statehood and nationalism are contested and are in the process of being renegotiated. Results presented here suggest that residents of Kazakhstan are acutely aware of national boundary traversals as they navigate the internet. The naming of a state-controlled space on the internet, through the use of ccTLDs, does in fact matter to the average user. Citizens of Kazakhstan often identified their activity on the internet as happening within or outside the space of the state to which they felt allegiance and attachment. We argue that naming matters for the creation of not only imagined communities online but also for individual expressions of nationalism on the internet.

Kazakhstan was previously part of the USSR.

There are several ways online spaces such as websites or other internet resources might signal their national affiliation. One such was is through the use of “country-code top-level domain names” (ccTLDs) that are in fact managed by an organization affiliated with the country in question that is the “designated manager” of second-level domain names (DNS) with the defined ccTLD (Postel 1994). The presence of a ccTLD often does not imply that the server that houses the page is in fact physically located on the territory of the country that the ccTLD denotes. However, symbolically, the webpage or an internet resource would display its national affiliation regardless of its actual physical location. We argue that the majority of internet users do not know and likely do not care where the resources they use online are physically located, but pay attention to the symbolic information embedded in the URLs as well as in the content they consume. In fact, prior research demonstrates that barring the physical locations of online resources, a direct analysis of links between sites based exclusively on their URLs indicated that most sites tend to link within a given ccTLD rather than across ccTLDs (Halavais 2000).

ccTLDs can operate as a national identity signifier, a way to entrench political borders on the web.

Although ccTLDs are the most common marker of national affiliation, they are rarely used in the US, suggesting a largely US-centric structure of generic TLD use such as .com, .net or .org (Leiner et al 2002). The lack of a country-identification for US businesses and personal sites may have been one of the drivers for the idea that the internet can be a borderless space. The use of ccTLDs is far more common in countries other than the US. We suggest that one of the reasons for this could be an attempt to carve out a national space on the internet where borders are deliniated, to clearly mark non-US territories and to provide symbolic markers for internet users.

The United States is an exception to this sort of identification. We’re the white people of the web in this way—the rarely-acknowledged default that doesn’t experience a national identification with our ccTLD because so much of the web (and our web) is US-built and US-centric.

Although much rhetoric in western countries still speaks of the one single internet that spans the world, the experience of talking about the internet in Kazakhstan begins to question this notion of a global undifferentiated online space.

Is there really a global single internet, or is there a series of differentiable internets that happen to use similar architecture and integrated infrastructure?

In Kazakhstan, many commented on the importance of both Russian and English for simply navigating online. For many young Kazakh-speaking respondents, however, use of Kazakh was an important marker of ethnic identity and a deliniation [sic] of national space online. Initially, young ethnic Kazakh activists translated interfaces of existing Western resources such as Facebook and Wordpress into Kazakh by contacting the companies and offering translation services for free.

Language matters too, as a way of communicating but also self-expression.

I recommend reading the whole thing.

Politics and server locations

Wed, 18 Oct 2017 17:49:36 +0000

Theorizing the Web 2014 included a panel on World Wide Web(s): Theorizing the Non-Western Web. The participants, from the program, follow:

Presider / Jillet Sarah Sam @JilletSarahSam
Hashmod / Alice Samson @theclubinternet
Panelists:
- David Peter Simon | @davidpetersimon | The Do-Gooder Industrial Complex?
- Jason Q. Ng | @jasonqng | Fit for Public Display: Rethinking censorship via a comparison of Chinese Wikipedia with Hudong and Baidu Baike
- Tolu Odumosu | @todumosu | Phoning the Web: A critical examination of Web infrastructure in Sub-Saharan Africa
- Dalia Othman |@daliaothman | Social Media, Activism and the Middle East

The live tweets from the session included some interesting tidbits.

Interesting to think about how the physical locations of Web servers fit into discussions of international politics. #TtW14 #b1

— Grace AfsariMamagani (@gafsari) April 25, 2014

These 7 popular Nigerian sites are all hosted in the US or Europe, causing latency. @TOdumosu in #Ttw14 #b1

— Cat Goodfellow (@catgoodfellow) April 25, 2014

“A truly indigenous web is of paramount importance” - Tolu Odumosu #TtW14 #b1

— Kira Simon-Kennedy (@sk_kira) April 25, 2014

Tolu Odumosu on the Nigerian web: it’s mostly mobile and even local sites are hosted in the US, slowing down connections #TtW14 #b1

— Kira Simon-Kennedy (@sk_kira) April 25, 2014

The borderless internet is a myth

Tue, 17 Oct 2017 17:48:55 +0000

The Atlantic, The Myth of a Borderless Internet. Political borders are re-enshrined on the web in a literal and metaphorical sense.

Just like the cartographers of yore, multinational corporations—particularly Internet companies—play a role in defining and shaping political boundaries for the public’s consumption. This rise of huge, international corporations online has torn away at the Emerald Curtain that once obscured the variety of geopolitical boundaries that exist in the world, making clearer to the average person just how unsettled the planet’s borders really are.

Given the global nature of the Internet, corporate giants like Google and Microsoft are forced to define borders, often contending with demands from governments. The result? One’s view of certain countries’ borders is often dependent on the physical location from which one accesses Google or Bing maps. In other cases—such as that of the Western Sahara—jurisdiction is a determining factor. Microsoft, which has offices in Morocco, takes its cue from Rabat in determining the territory’s borders, while Google—which does not—draws a dotted line between Morocco and the Western Sahara, demarcating the disputed border.

Political borders are enshrined in mapping tools, but reflected differently based on the nation-state that you occupy. The web has clear political borders, and the map you see on the web does too.

Rather than remove content entirely as other companies do, Twitter created a system whereby content would be “withheld” from users in a given country. Users are notified that the content in question has been withheld due to a legal request from a government. In addition to Pakistan, the tool has been used in numerous countries, including France, Brazil, and Russia.

The tool’s usage means that one “view” of the platform from a given country is different from the view from another. In other words, a Pakistani Twitter user is provided a sanitized version of Twitter, while an American one has access to—as far as we know—whatever content they desire. Corporate decisions around controversial speech, such as this one, all too often result in the creation of an “iron curtain” of sorts, dividing the seemingly borderless Internet.

The web you see in one country might not be the same web you see in another country. Political borders matter.

Who owns the ccTLDs?

Mon, 16 Oct 2017 17:47:09 +0000

Lawfare blog covers an interesting case that attempts to answer the question Are Top-level Domains Property?

On December 28 [2015], the Justice Department filed an amicus brief in Weinstein v. Islamic Republic of Iran, a case pending before the D.C. Circuit. At issue is whether country-code top-level domains are the property of those countries’ foreign governments.

Does a country’s government own the country-code top level domain that represents that country?

DOJ argues first that ccTLDs are not attachable “property” or “assets” under the FSIA or TRIA. Rather, ccTLDs “merely [] designat[e] . . . the national affiliation of a subset of the global Internet community,” including “millions of private businesses and individuals.

Although the right to designate its territory “Iran” is presumably valuable to the Iranian government, no one would suggest that the name “Iran” in an atlas or a newspaper—or even official publications—is itself the “property” of the Iranian government subject to attachment by creditors.

The Justice Department focuses primarily on the practical mechanisms of Internet governance. To support its position, DOJ points to a 1994 Internet governance document describing the Internet naming authority as a responsibility, not a property right, as well as “the actual practice under which country-code top-level domains have been established and managed.” In practice, ICANN “delegat[es]” TLD management to regional managers on the basis of whether the manager will be a “technically competent trustee of the domain on behalf of the national and global Internet communities.” In this sense, TLDs differ from second-level domains, which private parties purchase from the TLD-managers. Importantly, DOJ does treat second-level domains as property.

Internet domain stewardship is complex. Per the court:

any court order treating TLDs as property would threaten “the multi-stakeholder model of Internet governance” because other countries would react by “turn[ing] their backs on ICANN for good.” This risk of root zone anarchy not only eliminates any potential value for plaintiffs—who had hoped to profit from licensing Iran’s ccTLD—but also would “be devastating for ICANN.”

ICANN > a national government, in this case.

Searching without words

Fri, 13 Oct 2017 17:46:08 +0000

Search could be moving to images, in which case the languages may not play such a large part if images dominate web searches.

Fast Company goes Inside Baidu’s Plan To Beat Google By Taking Search Out Of The Text Era

In many cases, text-based search is not ideal for finding information. For instance, if you’re out shopping and spot a handbag you might like, it is far better to take a picture than to try and describe it in words. The same is often true if you see a flower or animal species that you would like to identify.

The Balkanization of the Internet

Thu, 12 Oct 2017 17:43:51 +0000

Political and legal borders interact to create a potentially balkanized future internet. Time Magazine says The Future of the Internet is Balkanization and Borders.

Rousseff’s plan to create walled-off, national Intranets followed reports that the United States has been surveilling Rousseff’s email, intercepting internal government communications, and spying on the country’s national oil company, so it was somewhat understandable. But her move could lead to a powerful backlash against an open Internet – one that would transform it from a global commons to a fractured patchwork severely limited by the political boundaries on a map.

The former Brazilian president wanted to protect her privacy by reinforcing political borders on the web.

The NSA has also opened a Pandora’s box by treating “citizens” and “foreigners” differently (even defining both groups in myriad different ways). U.S. rules also impose geo-locational-based jurisdictional mandates (based upon the route of your Internet traffic or the location of the data services and databases you use). Already, a German citizen accessing a New York City data center via a Chinese fiber line may find their data covered by an array of conflicting legal requirements requiring privacy and active surveillance at the same time.

What does it mean to be a citizen vs a foreigner when browsing the web and using the internet?

Chance thanks Obama for us with a ccTLD

Wed, 11 Oct 2017 22:42:18 +0000

Chance the Rapper has a new fashion line full of clothing that celebrates Barack Obama’s presidency: https://www.thankuobama.us/

With obvious references to Obama (the king Obama t-shirt) and some more oblique ones (a jersey with 44, because he was the 44th president), it’s only appropriate that the URL contains some symbolism.

He put the site up on thankuobama.us, indicating twofold:

Us as is “us” as in, we the people thank him for being our president.
But also Us as in US, as in USA, as in the ccTLD for the USA. The USA thanks him, but also like how much more patriotic of a URL can you get in this context.

Thank a former president with a URL that refers to the USA in multiple ways. Good work.

The tech media isn't flat

Tue, 10 Oct 2017 17:41:33 +0000

Model View Culture confronts “The App You’ve Never Heard Of”: Exploring Western Bias in Tech Media.

It is flabbergasting that LINE–an app that beats out Messenger and WhatsApp in Thailand and Indonesia–or WeChat or even Alibaba would ever be so baldly described as “little-known.” Little known to Americans or Europeans? Perhaps, since they were not part of the original target market. But “little known” to millions of people in Asia? Certainly not.

This pattern reflects the arrogance and shortsightedness of tech publications which, although often having primarily Western staff, are consumed globally: English speakers around the world — both within and outside of the tech industry — consume Western tech news; after all, Silicon Valley is home to international giants like Facebook, Apple, and Google. Such headlines erase huge populations of users, not only internationally but even in the West itself. Take, for example, the large population of immigrants to the West: just as WeChat remains significant to Chinese Australian immigrants and their families, many immigrants from non-Western cultural backgrounds remain connected to the technology of their (or their extended family’s) homeland. For example, South Korea’s most popular chat app, KakaoTalk, is installed on 93% of smartphones in the country; in America, the majority of KakaoTalk’s downloads are by Korean immigrants and Korean Americans. Not acknowledging just how significant KakaoTalk is to the Korean tech industry and to Korean Americans is exclusionary and, frankly, ignorant.

Just because some tech comes from Silicon Valley doesn’t mean that tech popular in non-Western markets is inherently “unknown.”

Yet a biased, narrow focus in tech journalism contradicts and subverts these outcomes. It’s time tech writers and bloggers educate themselves about what’s dominating the markets in parts of the non-Western globe, and move towards journalism that truly reflects a commitment to technology that is changing the world… not just Silicon Valley.

Less infrastructure, more control over the Internet

Mon, 09 Oct 2017 17:40:06 +0000

The less infrastructure investments and diversity of connections that a country has to the Internet, the more control they can exert over the country’s overall connectivity.

Most countries have gradually moved towards increased diversity in their Internet infrastructure over the last decade, especially as it concerns international connectivity to the global Internet. However, some countries remain at severe risk of Internet disconnection, with only one or two providers at their “international frontier”. This minimal diversity is often maintained for political purposes, making it easier to disable international Internet connectivity if deemed necessary

via Breaking the Internet: Swapping Backhoes for BGP | Dyn Blog

Sofar Sounds: So far from DIY

Sun, 08 Oct 2017 10:29:48 +0000

I attended my first Sofar Sounds show on Friday night. It was a great night, attending a show with a friend and making a few new friends with those that we were sharing a couch with. Sofar Sounds hosts shows in secret locations, from people’s houses, apartments, or even offices, that their community offers up.

As someone who went to a lot of DIY shows in college, the show was a bit surreal. The Sofar show had all the trappings of a DIY show: a crowd of people who care about music, who’ve all traveled there to see a show, a show organized by someone they know or another friend knows. Except in this case, the someone they know is instead a company that they’ve paid money to, and they don’t know the artists or the promoters or even the location until the day of the show.

The DIY shows I’ve been to were characterized by familiar faces, familiar locations, but also hardworking dedicated musicians and music lovers doing the promotion, organization, and crowd-wrangling. Shows in living rooms, basements, kitchens, and garages. Realizing years later that you should’ve worn earplugs. A network of people that once you break into, you can start going to even more shows in more and more places over the years as people move in or away.

The crux of it being, of course, breaking into the network. How do you find a community of like-minded people who have the resources to host and promote house shows, and are doing just that? Sofar Sounds takes people who have the resources to host house shows and connects them with bands and a predefined, curated audience. Sofar shows are like the Lyft of DIY shows, and the commercialization feels somewhat awkward. Sofar Sounds takes the DIY show model and tries to “solve” it with a business model.

Emma Silvers covers that business model in depth in her article A New Guest at Your House Show: The Middleman for KQED Arts. No longer do the musicians and music lovers have to do the promotion, organization, and crowd-wrangling on their own. Instead, they operate as volunteer “ambassadors” for Sofar Sounds, but don’t get paid and still have to do crowd-wrangling. The musicians, on the other hand, might not get paid anything at all. The audience is largely formed of strangers, selected based on applications for tickets by Sofar Sounds.

Perhaps because of all this, the community at the Sofar show felt constructed. Our “ambassador” made me feel like I was alternately at a sporting event or at a team-building exercise with his efforts to pump up the crowd and get us to bond with each other at the same time. Thankfully no one tried to fist bump me. We were all united in our love of music and our willingness to obey the rules about when we were supposed to leave or when we were supposed to talk. Overall, it felt distinctly constructed, rather than a true community of repeated faces like the DIY experimental/punk scene I’d known before.

At the show, I met some people that I enjoyed talking to and would want to see again. Therein lies another problem with the constructed community—it’s building a network behind Sofar Sounds, not behind the bands themselves. I may never see the people I met at the show again, because the network and the community of people attending the show is so far removed from those promoting and organizing it.

Nevertheless, the DIY network has limited reach, and Sofar might help bands break out of that network. If your DIY shows are always performed for your friends and your family, how do you attract new people? Building an organic community takes time, dedication, commitment, but doesn’t exactly pay you quickly. Sofar might act as a kind of shortcut to getting your music in front of new people that might not otherwise stumble into your community. Even if it also doesn’t pay you quickly, and even if they hear your one show and you never see them again.

What does this business model mean for local bars that host music? How many of the artists actually make money from the $15 cover that we’re willing to commit to this ~experience~? Perhaps next time I’ll spend my money at Hotel Utah Saloon or another local venue without the secrecy or the middleman. Maybe the next Sofar show should take you by surprise by happening at an existing venue, with local bands that get a full cut of the ticket cost. Of course, then we’re back at venues full of people that ostensibly don’t care about the music but rather the night they’re trying to have despite it.

As another alternate to Sofar Sounds, Group Muse also operates on the more commercialized house show model, but hearkens back to an even earlier method of hosting house shows—the era of chamber music. Capitalizing on the goodwill of hosts, the shows happen in the same sorts of venues as Sofar Sounds shows, and feature classical chamber music instead of more mainstream singer songwriter type music. However, musicians are paid by the audience, so the platform operates more realistically as a platform rather than a true middleman.

I’ve similarly been to one Group Muse event, and found it a great exposure to a type of music I wouldn’t have sought out otherwise. And that seems to be the real fun behind the secret show atmosphere. Someone has brought you to a place, you have already paid, and you have pretty low expectations of what you might be listening to. The openness that makes any sort of house show a success is already there. So overall, I can’t really complain too much.

The music industry has been disrupted a lot by technological advances, but artists continue to not get paid enough for what they do for us. So go to secret shows. Find house shows, find DIY shows, go to Sofar shows, go to Group Muses, and check out the acts playing at your local bar. Go out, dance hard, and pay up. It’s worth it.

Some artists I discovered at DIY shows:

Catalonia's Referendum and the Internet

Sat, 07 Oct 2017 18:38:32 +0000

Catalonia has its own top level domain, .cat. Not a vanity domain, this TLD provides an element of national identity in a region of Spain that has sought independence for many years. In the wake of the referendum, the office of the TLD registry was raided and computers seized. As reported by Internet News:

The Guardia Civil officers entered the .cat registry’s offices around 9am local time this morning and have seized all computers in the domain registry’s offices in downtown Barcelona.

The move comes a couple of days after a Spanish court ordered the domain registry to take down all .cat domain names being used by the upcoming Catalan referendum.

The .cat domain registry currently has over 100 thousand active domain names and in light of the actions taken by the Spanish government it’s unclear how the registry will continue to operate if their offices are effectively shutdown by the Spanish authorities. The seizure won’t impact live domain names or general day to day operations by registrars, as the registry backend is run by CORE and leverages global DNS infrastructure. However it is deeply worrying that the Spanish government’s actions would spill over onto an entire namespace.

A TLD is a symbol of national and political independence that the Spanish government is likely seeking to remove by taking this action. The Internet Society issued two statements about the raid, one initial statement and a second clarifying statement.

The initial statement made it clear that the Internet Society sees TLD operators as neutral parties:

We are concerned by reports that this court order would require a top-level domain (TLD) operator such as .CAT to begin to block “all domains that may contain any kind of information about the referendum”. We do not see it as the expertise and mandate of TLD operators within the Internet’s ecosystem to engage in monitoring and blocking of content outside of receiving judicial requests related to specific domains.

The second statement underscores this:

We firmly believe that intermediaries (in this case the top-level domain (TLD) operator, but it could be any other intermediary such as an Internet Service Provider (ISP)) should not be put in the position of having to decide what content is legal and what is not. Simply put, this is not the role of TLD registries.

Regardless of the fact that the Catalan region is not an independent nation from Spain (although they voted to become one in the referendum), the TLD helps the community identify as a distinct group with a distinct identity and community that deserves to be represented as such on the Internet. Wikipedia provides additional context about the granting and ideals behind the .cat TLD.

Universal language translation: thx, Google

Fri, 06 Oct 2017 17:57:37 +0000

Google is working on improving translation to the point where we have a universal language translator. In the Smithsonian Magazine, Kissing Language Barriers Goodbye:

“One thing that surprises people when we talk about Translate is our team doesn’t have any linguists on it,” Estelle says. “We’ve launched 71 languages, and I would say our team doesn’t know how to speak the vast majority of them. A human translator is not going to be able to learn all these terms and things as fast as our [data] can learn from the web.”

And with the recent release of their Pixel Buds, Google is trying to produce the sci-fi level real-time language translator. From Engadget’s article Google’s Pixel Buds translation will change the world:

Google packed its headphones with the power to translate between 40 languages, literally in real-time. The company has finally done what science fiction and countless Kickstarters have been promising us, but failing to deliver on, for years. This technology could fundamentally change how we communicate across the global community.

Real-time translation can do a lot for breaking down barriers. Engadget continues with the praise:

You’ll be able to walk up to nearly anybody in another country and be able to hold a fluid, natural language conversation without the need for pantomime and large hand gestures, or worry of offending with a mispronunciation. International commerce and communication could become as mundane as making a local phone call.

Universal translation might not be universally good, however. The Smithsonian Magazine article continues:

She thinks the device could be somewhat useful with travel, business and international relations but not groundbreaking. At a certain level, we already have translators (people) in place, and most who work in foreign relations know the appropriate languages. A device, Murphy believes, could have negative consequences.

“I think it can make people lazy,” Murphy says. Translating languages can be mentally challenging by forcing the brain—especially one that knows more than two languages—to work in a different way, but the exercise is rewarding, nonetheless. The brain pulls from a place of linguistic empathy that even the finest voice translator could never reach.

While this universal communication could be a positive, Murphy acknowledges, “it might lead to people thinking they’re communicating when they’re not.” Culture is not always completely embodied in language (take sarcasm, for example), and communication is not always about the information being passed.

Translation is the “key” to unlocking a “truly global” web, but this sort of translation could also lead people to think they’re communicating when they’re not (to paraphrase the article). If you think you’re communicating with someone, but you’re talking past each other in a series of miscommunications, the UX of your world is off, so to speak. This happens in same-language communication as well. With the additional layer of auto-translation interference, it might be easier or harder to detect when such miscommunications happen.

Borders on the Web Series

Thu, 05 Oct 2017 22:34:34 +0000

The web is sometimes spoken of as a borderless place. Through the ~magic~ of technology, the internet, the “decentralized” web, borders would be eliminated and the world would become truly flat.

I’ll share links as part of this series that reinforce and challenge that notion.

Linguistic borders, reinforced by lack of multilingualism yet challenged by machine translation successes.
Geographic borders, reinforced by internet infrastructure yet challenged by novel methods of providing internet access.
Legal borders, reinforced by censorship and international agreements yet challenged by citizens and corporations.
National/political borders, reinforced by internet service providers, top level domain names, and governments yet challenged again by citizens and corporations.

Data as a Gift: Implications for Product Design

Tue, 22 Aug 2017 18:22:20 +0000

The idea of data as a gift, and the act of sharing data as an exchange of a gift, has data ethics and privacy implications for product and service design. Recent work by Kadija Ferryman and Nick Seaver on data as a gift in the last year addressed this concept more broadly and brought it to my attention. Ferryman, in her piece Reframing Data as a Gift, took the angle of data sharing in the context of health data and open data policies. Seaver, in his piece Return of the Gift, approached it from the angle of the gift economy and big data. Both make great points that are relevant in the context of data collection and ethics, especially as it relates to data security and privacy more generally. Ferryman introduces the concept brilliantly:

What happens when we think about data as a gift? Well, first, we move away from thinking about data in the usual way, as a thing, as a repository of information and begin to think of it as an action. Second, we see that there is an obligation to give back, or reciprocate when data is given. And third, we can imagine that giving a lot of data has the potential to create tension.

When you frame the information that we “voluntarily” share with services as a gift, the dynamics of the exchange shift. We can’t truly share data with digital services—that implies that we retain ultimate ownership over the data. You can take back something after you share it with them. But you can’t do that with your personal data.

Because you can’t take back your data after you share it, you can more accurately conceptualize the exchange of data with digital services as a gift. Something you give, and which cannot be returned to you (at least not in its original form). Data as a gift creates an expectation or obligation for a return, Seaver makes clear. Problem is, when we’re sharing data on the internet, we don’t always know exactly what we’re giving and what we’re getting.

The gift exchange might be based on the expectation that your data is used to provide the service to you. And the more data, the better the service (you might expect). For this reason, it seems easier to share specific types of data with specific services. For example, it’s easier for me to answer questions about my communication or sexual preferences with a company if I think I’m going to get a boyfriend out of the exchange, and sharing that data might make it more likely.

But what happens if a company stops seeing (or doesn’t ever see) an exchange of data as a gift exchange, and starts using the data you gift it for whatever it wants in order to make a profit? By violating the terms of the gift exchange, the company violates the implicit social contract you made with the company when you gifted your data. This is where privacy comes in. Gifting information for one purpose and having it used for other unexpected purposes feels like a violation of privacy. Because it is. A violation of the gift exchange of data is a privacy violation, but it feels like the norm now.

It’s common in terms of services to be informed that after you gift your data to a service, it is no longer yours and the company can do with it what it wants. Products and services are designed so that you can’t pay for them even if you want to. You must share certain amounts of data, and if you don’t, the product doesn’t work. As Andrew Lewis put it:

“If you are not paying for it, you’re not the customer; you’re the product being sold.”

We didn’t end up there because we are that dedicated to free things on the Internet. We were lured into gifting our data in exchange for specific, limited services, and the companies realized later that the data was the profitable part of the exchange. Nick Seaver refers to this as “The obligation to give one’s data in exchange for the use of “free” services,” and it is indeed an obligation.

To avoid gifting your data to services that you might not want to enter into that type of exchange, you have very few ways to interact with the modern Internet. You’d likely also have to have a lot of money, in order to enter into a paid transaction rather than a gift exchange with a company in return for services. For those of us working in product or service development, we can use this perspective and consider the social contract of the exchange of data gifts.

Consider whether the service you offer is on par with the amount of data you ask people to gift to you.
- Do I really need to share my Facebook likes with Tinder to get a superior match?
Consider whether the service you offer can deliver on the obligations and expectations created by the gift exchange.
- Is your service rewarding enough and trustworthy enough to where I’ll save my credit card information?
Consider whether you can design your service to allow people to choose the data that they want to gift to you.
- What is the minimum-possible data gift that a person could exchange with your service, and still feel as though their gift was reciprocated?
Consider the type of gift exchange that you design if you force people to gift you a specific type or amount of data.
- Is that an expectation or obligation that you want to create?

When you view each piece of information that a person shares with you as a gift, it’s harder to misuse that information.

Note: Thanks to Clive Thompson for bringing Kadija Ferryman’s piece to my attention, and Nick Seaver for sharing his piece Return of the Gift with me on Twitter.

Feature Names Matter

Fri, 11 Aug 2017 11:28:35 +0000

When someone starts using your software, they need to build an understanding of how it works and how the pieces interact. The UI text you write and the feature names you choose can build or break a mental model.

From a marketing perspective, the importance of the name is clear. You want something catchy, marketable, searchable, memorable, all these things. But most importantly, a feature name must help a user build a mental model of what your feature does.

The mental model helps the user understand why they might use this feature, and what for. One of the riskiest part of shipping something new is adoption. If people don’t know what it does or how it works, they won’t use it. A crucial element to that understanding is what you call the new thing and how you describe it in the product. If I can’t guess based on the name what it does, I might not click on it at all and explore it.

Let’s look at some feature names…

Google+ vs GoogleDocs. One of these is pretty opaque, and the other is pretty clear. I might think that Google Docs is google FOR docs, but as soon as I click into it, I’ll see what it is and understand that it’s for writing docs. I might never click into Google+ because I have no idea what it is based on the name.
Dropbox vs Box. There’s a reason both of these companies are named practically the same thing. Because you put things in boxes that you want to share and store. It’s a super evocative mental model, so it gets a bit overused, perhaps.
Slack vs HipChat. HipChat is a bit more descriptive, but you know automatically that it’s a chat app. Slack turns a verb into a noun, and hopes that you start using it and understand that you slack off while using it… kind of.

It’s harder to come up with examples in software of things that truly failed, because they aren’t very well known. But the example that brought this to life for me is from a card game I learned how to play recently. Red7 uses the concept of a “canvas” and a “palette” to tie the metaphor of color across the game. But combining those concepts with the established mental model that you have in a card game with a discard pile and a hand of cards took quite a bit of work. In reality, the clever metaphor broke down and impeded what could have been quick understanding by burdening an existing card game mental model with a mental model of painting ephemera. It was marketable, but not intuitive because it didn’t help people build a mental model to understand how the game works.

The simplest way to pick a good feature name is to test them out. Do some word association exercises with your team, but also with people that don’t work on your team and don’t even work in software. Diverse teams matter a lot in this exercise. This can help identify names that build mental models, break them, or are irrevocably associated with irrelevant mental models.

Another way to pick good feature names is to rely on scenarios when building features. That way, you’re less likely to conceptualize a feature based on its architecture, or your internal team structures, and more likely to think of it from a problem-solving perspective. If you know exactly what the feature is doing, and for whom, it’s easier to pick a useful name.

My impressions from the 2017 Bay Area Book Festival

Sun, 04 Jun 2017 15:15:24 +0000

For the second year in a row I attended the Bay Area Book Festival. A collection of authors, publishers, readers, and other bookish sorts also show up for the weekend in Berkeley. This year, like last year, I discovered some great sessions that led me to think about things from a new perspective and that I might not otherwise have learned about.

One great thing about the book festival is that, true to its slogan, it connects readers and writers. Much like the podcast Song Exploder, the sessions caused me to think far more about the process behind creating the things I love than I might have otherwise.

Something that struck me and my friend, however, is how few of the attendees were young. There are a lot of attendees (of sessions, especially) that seem to be retired, quite a few middle-aged people, and in the YA-fiction sessions I attended, a good number of children and their parents. In fact, much of the festival seemed designed to be family friendly.

However, in a reflection of publishing at large (perhaps) the festival neglects the population that is most clearly present in Berkeley and the Bay Area to me—the ~ * millennials *~ that attend the university and work in tech and other industries all around the bay. I saw a few other people in their mid-twenties and mid-thirties, but not nearly as many as I’d hope to see. No one that I talked to in that age range (save my friend) knew about the festival that weekend, and I saw almost no marketing for it aside from the emails from the organizers (thanks to signing up for the email list last year) in the social media sites and feeds that I follow. Hopefully in future years this will change.

I also struggled to determine who the actual target audience of the festival was based on the sessions I attended and the booths I witnessed wandering around (I’m not much of an actual booth-visitor). The fest again, advertises itself as connecting readers with writers, but it seemed to be connecting writers with other writers just as often. Whether on panel discussions or through writer-oriented booths in the park, that was a very present and understandable theme. I’m not sure it matters that the target of the festival is so broad, and perhaps it might flourish even more if it were more explicitly targeted to this wider audience. But again, perhaps not.

I attended two sessions each day of the festival, and here are my impressions of each of them.

Saturday

The first session I attended on Saturday was called Worlds We Create: Young Adult Fantasy Writers on Creating Alternate Realities and Memorable Characters. It’s only in the last year or so that I started realizing that I do read fantasy and sci-fi novels, and while I didn’t seek them out when I was younger (sorry, Tamora Pierce and Phillip Pullman), I always enjoyed historical fiction and fairy tale retellings. This session was explicitly about how fantasy writers go about creating the worlds in their novels. The moderator had some great questions, they asked about how authors develop characters to reflect the world they inhabit, whether the story or the characters or the world come first, and more.

They discussed using “sensitivity readers,” a concept I hadn’t heard of before this panel, who you can hire to provide feedback and criticism about the way that you’ve told the story of a character with experiences that you haven’t had directly. One of the authors, who set her latest book in turn of the century New York City, spoke of the balance of being sensitive to certain life experiences and having to confront discrimination with the focus of blowing it up in the book.

Needless to say, I’ll be thinking a lot more about the quality of worldbuilding after this panel. I was in the midst of reading a series that was set in a world that took awhile for me to understand, and seemed to use a glossary to attempt to offload some of the worldbuilding heavy lifting, yet the story and the characters were fairly well-developed enough for me to appreciate it and continue following through. The balance of world-building time and story-telling time was also covered in the panel, with one of the authors mentioning that she’ll use worldbuilding as a way to get started writing a new book if she’s struggling with how to start.

The second session I attended on Saturday was rather different from the YA fiction panel. This session featured Masha Gessen in conversation with Orville Schell on Truth, Lies and Totalitarianism in Russia and the U.S.. This session featured quite a few great moments. My friend pointed out that she hadn’t before heard someone speak with such attention to the precision of their language, yet that is exactly how Gessen spoke. They were very precise when agreeing or disagreeing with Schell, and didn’t mince words. Some paraphrases of what they said in the session, along with my own characterizations about what they said, follow.

Totalitarianism succeeds through destruction… not durability of the presence of totalitarianism or genetic makeup or expectations but of the absence of shared understanding and society and other things that help you move forward in the face of it.

They took care to distinguish her perspective from that of Schell’s. He asked a question about whether totalitarianism or autocracy persisted in some countries due to the genetic makeup of people or due to a durability of the people, and they disagreed, though recognized that they agreed on the state but not the cause of the state. The absence of certain things, rather than the presence of something, is what is helping totalitarianism succeed and persist in these nations.

We’ve constructed a story about what happened in post-soviet states but not what happened in the west… democracy isn’t a state achieved by a country but a spectrum (becoming more or less democratic)

I really appreciated that they mentioned this. As someone who studied post-Soviet states in college, I am quite familiar with the stories constructed there, but I hadn’t considered this perspective until they mentioned it. In my recent travels and events in the United States, it’s become clear that the Western states didn’t pay much attention to developing an identity that didn’t position other states as enemies and actively maintaining democracy.

Gessen also mentioned, perhaps in response to an audience question, why Trump admired Putin so much. In response to that, they referenced Tim Snyder in the NYRB, who wrote “Putin is the person that Trump plays on TV.” Expanding on that point, they continued that “Trump talks of power, off the charts popularity, and control… but doesn’t really understand where that comes from.” And where that comes from is autocracy. They also discussed their own article in the NYRB about autocracy. Also discussed was the differing perspectives of politicians, businessmen, and others in an autocratic state compared with a democratic state.

There’s this belief from Russian politicians, journalists, businessmen that everything can be bought or is for sale or is transactional… a few years ago that didn’t fly here but that’s changing here… apparent struggle to understand non-transactional motives in authoritarian societies and possible explanation for a lack of understanding of civil society… If public service and nonprofit motives are questioned or not understood then democracy is challenged

Schell also brought up countries that don’t deal with history in an honest way (like many totalitarian states, Russia or China especially have quite censored and limited views of their own past), Gessen spoke of WWII as being a touchstone in Soviet history in rewriting the past, but also continued to discuss whether a society can move forward with a distorted view of the past.

All societies have distorted views of the past…that’s trauma that societies have to deal with. Estonia for example has done a good job with this: “we trust one another, we were under occupation, and we’re going to rebuild based on that trust.” Germany had the advantage of short period of time and everyone could remember pre-totalitarianism and talk about perpetrators, bystanders, and victims…. but it’s totally muddled in the Soviet Union; people were both perpetrators and victims and no one was a bystander. How do you take responsibility for your own past if you messed it up? Whitewash the history…

One of the things that they mentioned as being most challenging for rebuilding in (or after) a totalitarian or autocratic state was imagination.

How do you rebuild when you can’t imagine a future or a past…and sometimes you can’t imagine the present because it’s so awful that you can’t fathom it…you reach for the past and confuse lack of imagination with understanding

When asked about her hopes for Russia in the future, they were blunt. “Russia is hopeless, scorched earth.” They did expand in response to a follow-up question to discuss the small-scale successes of some friends of theirs in starting focused community organizations, but acknowledged that people seeking to make change on a small scale, while inspiring, is a way of admitting defeat. After you start focusing almost exclusively on small-scale change, you’re no longer trying or dreaming of making those changes on a large scale.

Sunday

Sunday I slept in, so I skipped an earlier morning session I considered attending so again attended two sessions.

First I attended another YA fiction session, Reality Bites: Fiction About Teens’ Real Lives, which was really great. Another panel discussion, the authors discussed what they do when they write most of a book and realize it isn’t working, sometimes all you can do is “upcycle” some of the scenes into other books. The moderator also asked about recurring themes and iconography, and while one author mentioned trees (she writes largely thrillers that tend to involve the woods), another author mentioned that she often writes about the sky, likely because it represents us against the vastness.

Most relevant of all, I thought, was the discussion on how writing novels teaches you to be compassionate about multiple points of view. Kim Culbertson especially made some great points about this. Sometimes a character has to do something differently than the way you would. She mentioned that “Lots of people are uncomfortable when they hold themselves up to another person and the edges don’t match.” But she likes to challenge that in a way, to help people get to the understanding and consideration that other people are different from you and that may seem strange but it’s actually awesome, and we need to recognize that!

The second panel I attended was about immigration and identity, Living in Two Worlds: Crossing Borders and Identities to Create Home. The moderator asked each author in turn to speak about their book which led to a great focused discussion on the themes in each one.

First, Laleh Khadivi discussed her book and the challenge about writing about two worlds, especially that “one world is never going to believe the other world.” Next, Lesley Nneka Arimah discussed her book of short stories. She mentioned that she included Facebook in one of her stories because she was tired of reading stories set in contemporary Africa that didn’t mention social media. She underscored that while traditional infrastructure in Nigeria and other countries may be lacking, everyone has a cell phone and electronic communication is crucial to how they live their lives. It seemed out of touch not to acknowledge that reality of life in her fiction.

Carolina de Robertis spoke next about her novel, but my favorite quote by her came from the Q&A after the moderated panel concluded, which was that “Culture is not static; we can shift culture and push it open with the narratives that we live and write.” Pajtim Statovci spoke last about his novel, and addressing the fact that living through the trauma of immigration, especially as a refugee, doesn’t automatically make you stronger.

Sometimes our negative experiences don’t make us stronger. Sometimes they make us weaker and sad and pathetic.

That resonated with me, because it’s easy to look at those who have survived something traumatic and assign strength to them merely because they survived. But that isn’t always the case. This thread resonated with the other books on the panel as well, because the main character in Khadivi’s book is an immigrant who radicalizes after immigration. Lesley Nneka Arimah mentioned about wanting to write about flawed characters as well, in the vein that we often think of ourselves as better than we are, so she wanted to write about characters that do the wrong thing in service of finding their own truth.

Overall impressions

The festival was worth my time to attend, and spend time thinking and talking about books with and around other book-minded people. I look forward to the next year’s book festival and the sessions! It’s always easy to attend when it’s convenient to get to and the weather is beautiful. Now I’m inspired to tackle even more of the books on my shelves…

Tips for live tweeting an event

Mon, 20 Feb 2017 19:12:39 +0000

If you use Twitter and are attending an event that you want to share with your twitter followers, you can live tweet it as it’s happening. While you can live tweet basically any event, these tips focus mainly on talks that you might attend as part of a conference, a meetup, a sponsored speaker series, or another presentation. I’ve live tweeted several conferences (two as part of a job, such as #SUMIT14), talks, and series of talks as @smorewithface. First, the basics on live tweeting an event, then some pro tips and best practices to follow before and during the event.

Tweet! Tweet live!

My tips for live tweeting an event.

Type fast. It is live, after all!
Don’t tweet everything. If you tweet everything the speaker says, you’ll overwhelm the feed and you won’t ingest much of the information yourself. Be prepared to decide on the fly when to quote, when to paraphrase, and when to summarize several key points in the same tweet. Don’t think too hard about it, and don’t be afraid to stop tweeting until you hear something that strikes you as interesting or relevant.
Consistently use the hashtag. If the event has a hashtag, use it every time. And I mean every time. If you forget, delete your tweet and retweet it with the hashtag. This helps your followers who might have muted your hashtag, and it helps others interested in the event find your tweets. If there isn’t a hashtag for the event, don’t be afraid to start your own!
@ mention the speaker. If the speaker is on twitter, @ mention them if you quote them or are introducing them at the beginning of the talk. Don’t mention them in every tweet—if they want to see everything you said, they can click through to your timeline.
Add visuals. When it makes sense, share photos or other visuals. Share a photo of the speaker to start off your tweets, or a snapshot (or better yet, a screenshot) of an especially complex or pithy slide. Crop them so the most important parts appear in the preview. Don’t tweet blurry photos. Retake or move on.
Include crowd reactions. Sharing how the audience responds to key points made by the speaker (applause, laughter, gasps) can add context and depth to your tweets. Stick around for the Q&A to share questions that people asked and emphasize parts of the talk that resonated with others in the audience.
Share personal speculations. React in real-time and speculate about how the content of the talk resonates with you.
Relate to other events. No talk is unique. Feel free to add context by calling attention to similar approaches or alternative viewpoints in talks at other events—other talks at the same conference, talks at a similar conference, talks that happened earlier in a series, or upcoming talks.
Share articles and other content. Be quick with searching, or prep beforehand and share links to articles, books, and other content that the speaker references in their talk.

With all of those tips in mind, here’s how to be more prepared to follow through in the heat of the tweeting.

Before you start

Before you go to the event, consider these questions.

Who are you tweeting as?
- If you’re tweeting as yourself, a live tweet of a talk reflects your personality, interests, and expertise. You also might want to warn your followers (though hopefully if they’re following you, they’d be interested in the subject of the tweets!)
- If you’re tweeting professionally as the event, or an organizer of the event, you likely want to tweet less often. Stick to the facts when tweeting, and don’t editorialize as much. You’ll also want to pay attention to attendees of the event that are tweeting, and retweet the good tweets among them. It also depends on who is likely to be following the account that you’re tweeting as. You don’t want live tweets to crowd out essential announcements or logistical information.
Who are you tweeting for?
- If you are tweeting to the backchannel, the people attending or viewing the same event as you, you can tweet more reactions to the content rather than quotations or paraphrases of the content. A good example is the @dog_rates account live tweeting the puppybowl.
- If you’re tweeting to people who couldn’t be at the event, but would otherwise be interested, you’ll want to share more of what the speakers are saying, in addition to any reactions you or the audience might have. I tend toward this style of livetweeting.

Some additional considerations before the event include some extra preparation. Especially if you’re tweeting in an official context, these tips can take your tweet game to the next level.

Look up the speakers. Some speakers remember to add their twitter handles to their slides, others don’t. Be prepared and look it up beforehand. Also, many speakers are giving talks based on recent work that they’ve done. Look at their personal websites for articles, books, or other recent content that they’re likely to reference in their talk.
Get familiar with the topic. You might not have to take this step if you’re attending an event due to your personal interest, but you might not be. Do some basic reading on the topic before the event to get familiar with some of the jargon, references, and “big players” in the field (be they people, companies, or organizations) so that you can accurately refer to them on-the-fly.

Recommended setup

After you know what you’re doing, get prepared to successfully live tweet an event.

Use a laptop. A phone will do for casual live tweeting, but if you want to go all-in, bring your laptop.
Plan your power. Pay attention to battery life and your seat location if you plan to be live tweeting all day.
Get on the network. You need internet to tweet. Get it, somehow.
Sit politely. You’ll be typing pretty constantly throughout a talk. Don’t sit near a mic, and don’t sit near people that you might bother. Try not to type loudly if you can help it (I can’t). Be attentive to people that may be looking at you often that don’t say anything. You’re either bothering them, or they think you’re not paying attention and you’re lame. They can think what they want about your intentions, but try not to intrude on their experience of the event.
Join the videoconference. If the event is being livestreamed, join the videoconference. Only do this if the slides are being shared, and if the network connection at the event will withstand it. The only reason to do this is to get classy screenshots to share with your tweets.
Type in a text editor. You can type all your tweets in Twitter, but you might end up typing a tweet at the same time that you want to start writing the next one. If you type in a text editor, you can take notes for a second tweet while you finish typing the first one. This is also helpful for “sum up” tweets that don’t quote the speaker. You can also better concentrate on recording the content first and worrying about character count second. You can also have your prep notes about the speaker(s) and links to their recent work in the text editor for quick and easy reference.
Have a search engine open. If you want to quickly pull up a reference to a concept, book, paper, event, artwork, etc., you might as well save a few seconds by having a search engine already open.
Consider a client. If you want, use a twitter client like Hootsuite instead of Twitter. The advantage to this is that it is easier to have multiple feeds open at the same time, so you can monitor the other tweets shared by other attendees using a search on the event hashtag, have your own profile page open, and view your twitter feed all at the same time.

Caveats!

Live tweeting isn’t for everyone. You can enjoy an event without Twitter. You might actually remember more! You can review your tweets later as a form of notes, but you wrote them to other people so you may prefer to take your own notes to record your thoughts instead. The hive mind can be valuable, but also oppressive. I often bow out of live tweeting an event and savor it instead. Some events are also difficult to live tweet and share in a text-mode or with others that aren’t there. Don’t force it for the notification love. Live tweet for you (or your job).

Hey but what about liveblogging, live chatting, live slacking, live…

In my opinion, live tweeting is slightly different from those media. A liveblog has an interested audience, there for all of it. They are likely following a liveblog in lieu of attending the event, and want to know much more detail about what is happening. Live tweeting involves your followers, who are there to follow you and not necessarily a barrage chronicle of an event. You want to curate more on Twitter. You want someone to get the same insights that they might have gotten if they had attended the same talk or conference, but as told by you, rather than every single insight. There is room in liveblogging for personal reactions and commentary, but I think the audiences and the goals of liveblogging and livetweeting are somewhat different.

2015 Resolutions, 2016 Music, and 2017

Wed, 04 Jan 2017 19:55:02 +0000

In 2015 I made some resolutions. I haven’t followed up on them since. In 2016 I made no resolutions, but I listened to a lot of music.

How the 2015 resolutions fared in 2015 and 2016

I did okay.

1. Stay off Twitter more, read fewer articles on the web, and create more

I’ve continued to use Twitter over the past couple years. My use of it waxes and wanes depending on the news. I periodically delete it from my phone to get a break. My Pocket queue and a change in my commute mean that I’ve certainly shifted my web-based reading habits. I read 73 books in 2016 (that I bothered to add to Goodreads, so let’s round to 75). I’ve taken a few vacations without my laptop, and have spent a bit more time working with my hands—whether at the gym, climbing, or making things like jewelry.

2. Learn JavaScript

I still have not learned JavaScript. I gave up on it. Part of this is because semicolons are rude, and part of this is because I joined a company where Python is the primary backend language. I realized Python would be easier to learn and my interests shifted more toward data analysis (and maybe some digital mapping?) and further from interactive web content, so I worked on learning Python instead. I ran into the same issue learning Python as I did JS though… it requires a lot of time, and a lot of practice. Side projects are hard to maintain, especially when they’re similar to your day job. I still have some tabs open about learning Python, and I took more Python tutorials, so I have at least reading-level knowledge of both languages.

3. Read something huge, and write something huge

Still haven’t read Gödel Escher Bach: An Eternal Golden Braid. Still haven’t written up a magnum opus on the reification of geographical and political and linguistic borders on the web. I’m hoping to break it apart and start publishing bits and pieces here this year. We’ll see.

Goals for 2017

I’m meeting with friends this weekend to work through some. I’m setting some in terms of reading material, to better improve my knowledge of key (also depressing) moments in history. I can’t be a history nerd without being fascinated by dictators, oppression, and systemic discrimination. So I’m branching out to different geographical dictators and oppression in history this year. Beyond that, we’ll see what goals I end up with.

2016 in Music

2016 was a music-filled year, after a multi-year hiatus of drifting away from it being as central in my life as it deserves. These songs stuck with me for various reasons. They helped me regain some hipster credibility, branch out into less-explored genres, and reminded me how important it is to support artists you care about.

Songs that stuck

Gibbz - Love Again
Novo Amor and Ed Tullett - Alps
Ella Vos - Little Brother
Alle Farben - Please Tell Rosie
Daughter - No Care
The xx - On Hold
- I bought the album before it came out based on the strength of this song alone.
Sebastian - Embody
Run the Jewels - Talk To Me
DJ Shadow - Nobody Speak (feat. Run the Jewels)
Bon Iver - 22 (OVER S∞∞N)
Justice - Alakazam!
Miike Snow - Heart is Full (ft. Run the Jewels)
Sjowgren - Seventeen
Slow Club - Rebecca Casanova
Just Jack - Lonely Telephone

This year I bought the CHVRCHES album, the Bon Iver album, the Frank Ocean album, and The Little Prince soundtrack. In addition to a bunch of one-off songs, because shuffle is how I roll. I listened to Jason Derulo a lot, and listened to a four song playlist of Carly Rae Jepsen, Ariana Grande, Ingrid Michaelson, and Adele for longer than I’d like to admit.

Shows I saw

I saw the following artists play live:

Justice and Sebastian (he might still go by SebastiAn)
Broods and Two Door Cinema Club
Daughter
Still Flyin’, Annie Hart of Au Revoir Simone, and Slow Club
Tallest Man on Earth and The Head and the Heart
Cold War Kids
CHVRCHES

If I had to rank them, CHVRCHES would be first. I started the new year by binging on SoundCloud recommendations from The Burning Ear and the related tracks that SoundCloud plays. A great way to fall into a rabbit hole of discovery, and a totally different experience from Spotify’s more carefully-constructed and curated experience of the Discover Weekly and other playlists.

Stats that were gathered for me, passively by last.fm

Last.FM tells me things about last year in music too:

My listening increased after I moved in August.

CHVRCHES was a continual favorite.

I listened to a lot of different types of music, but mostly stuck to indie.

Hey Rosetta! - Trish’s Song is a great song to listen to if you’re trying to fall asleep.

2017 in Music

In 2017 I have tickets to see these artists in concert…

Less Than Jake
Mike Doughty and Wheatus
Jens Lekman
Radical Face
Matt Pond PA
Gibbz
Knox Hamilton and Colony House

And that’s just the first three months. City life suits me. Forging ahead into 2017 suits me. Here’s to more reading, more music, more learning, more blogging, and more self care in the year ahead.

Netflix, Amazon, and Hulu: Watching and Sharing on the Web

Mon, 02 Jan 2017 19:43:27 +0000

I like to share with my friends. I’ll send them links to articles in Pocket, book recommendations on Goodreads, and music recommendations in Spotify. I’ll also post SoundCloud songs on Facebook, quote articles on Twitter, and email other articles to friends. All of these sharing methods augment word-of-mouth, letting me seamlessly share my experiences with others within or outside of an application.

What’s missing is the ability to easily share recommendations in the apps we use to watch TV and movies online. The word-of-mouth augmentation that social media provides is pretty minimal when it comes to TV and movies.

You can ask your friends for recommendations about what to watch (or you’ll see their posts online about what they’re watching) and end up with a list of shows or movies to check out. But keeping track of what you want to watch, and where you can watch those shows and movies, isn’t easy.

It would be easier to directly link to or share a Netflix-only series or movie with someone else. Click and watch.

I started using Spotify because it was so easy to share songs with people. SoundCloud didn’t have everything that I wanted to listen to (and my more casual music-listening friends didn’t use it), listening to music on YouTube always sucks, so I turned to Spotify.

It’s so low friction to share songs with other people on the service that it’s supplanted iTunes and Google Play and SoundCloud and even Youtube for native sharing for me. I want to be friends with people on Spotify just so I can send them music. It’s easier to craft a playlist and share a link than it is to send a mix CD to someone nowadays.

In the digital ownerless economy that we’re encouraging, we still want to share our discoveries and spread our interests far and wide…but when you don’t own something, you can’t share it as tangibly as before. You can’t send someone a DVD of a movie you think they should see, you can’t tell them to watch NBC at 9pm on Thursday night—because you may not own a DVD player, let alone any DVDs, and you definitely don’t have cable anymore.

But it’s just as hard to send someone a recommendation to something on Netflix or Hulu. The best way to recommend something to someone is still to say “Hey! Do you have Netflix? Watch Stranger Things!” or “Hey! Do you have Hulu? Watch The Bridge!”

Maybe that’s working for now. Maybe that kind of word of mouth helps raise the subscriber count for those services—being constantly asked if you have Netflix or Hulu might make you more inclined to sign up.

But what really got me using Spotify, not just opening it, is the social aspect. You log in and you can see the person- or algorithmically-curated playlists of things you might want to listen to, but those recommendations are often no substitute for a recommendation from a friend. When subscribers of your service can easily share what they enjoy with other people, you don’t just have subscribers, you have happier subscribers.

Netflix offers you the “Suggestions for you” based on things that you’ve already watched. But Netflix can’t know you. It doesn’t know what you watch and like outside of its service and offerings, and there’s no way to supplement the algorithmic suggestions with people-curated suggestions from people that know you.

As the internet exposes us to a plethora of things, sharing and searching become a crowded market. In a stream of noise, we turn to services that help us carve out enclaves of interests. Netflix thrives as a primary place to watch movies and television. Spotify thrives as a primary place to listen to music. The names are often temporary and shifting, but what gives them staying power is the ability to interact and discover new and intriguing things within our interests.

But discovery is hard.

It’s harder still to find the things that are locked down in walled gardens. I can’t search across Netflix, Amazon, and Hulu to see which one might have a movie that I want to watch. I have to check each site individually. This is a hassle for people who want to track down a recommendation. You can’t always do a web search for the show or movie name (or if you do, you might be more likely to find a bootleg site rather than a legitimate service you can use to watch it).

If you’re a group of movie and television buffs that don’t live near each other, you can’t easily share what you’ve watched and what you think others should watch. You have to choose an alternate interaction media—email, Facebook, Twitter—rather than share your recommendations and history of what you’ve watched within the service.

A simple option is using the permalinks built in to each service to offer a sharing option. Hulu already does this by letting you one-click share to Facebook or Twitter from the title page of a movie or show, but Netflix does nothing to encourage you to share with others. Neither service lets you share a recommendation directly with other users of the service.

Amazon is the dark horse in the streaming services. It strives to have a listing for basically everything in existence for sale (as far as I can tell), so it’s easy to link to just about anything you might recommend someone watch. Even if you can’t stream a show on Amazon, you can often buy the DVD, rent it, or add it to a list. Amazon isn’t the go-to streaming service for most people, but you can send a link to just about any show or movie from Amazon. Still, though—there is no way to share a direct recommendation to other Amazon users.

It’s understandable that none of the services have a native social option yet. A native social option is hard to build up and maintain, from both the people and engineering aspect.

Goodreads has a full-fledged social network that is underused (if used at all) by the users of its service. Spotify has a more passive option that would work well for TV and movie watching. On Spotify you can share what you’re listening to (with an optional private mode for the guilty pleasure songs) and you can send a message to another user that you’re friends with in Spotify or externally to apps like Facebook or Twitter.

Building something like this would have challenges.

Abuse is an obvious one. Any place you can interact with other people means that you might be subject to harassment or abuse. Any site that permits interaction with other people must also provide the digital equivalent of walking away from one person or closing the door in the face of a flood.

Adding social interaction to an already complex system of scaling, encoding, and transmitting high-quality video across the globe wouldn’t be easy either. How do you take a private account and add a social component? How do you find the accounts of friends to connect with? Would Facebook end up the most common denominator, a de facto social address book?

To obviate these issues, start simple. Add a share sheet to the pages that display before and after you watch a show or a movie to let people share what they want to watch, or just watched. Amazon has a share option, Hulu lets you share to Facebook or Twitter, and all three services have links that go directly to the show or movie within the service.

Track how people interact with those links, and make further decisions based on that interaction (and of course, by talking to users). If the links are most often shared on Facebook, an integration with that service might be a first step toward social. If links are more often sent in messages or emails, accessing a person’s contact list might be a better first step.

Native sharing in services like Hulu or Netflix wouldn’t necessarily be easy to implement. But you’d gain the opportunity to easily send your broken-hearted best friend the movie that helped you get over a heartbreak. Your sister can send you the TV show she thinks you’d like, but that you keep forgetting to look up when you get to your computer. You’d regain the connection to far-flung friends that sharing a common experience can give you.

Physical technology and inherited meeting places

Sun, 11 Dec 2016 18:47:19 +0000

Digital technology is central in our social and personal lives. Laptops, smartphones, and the apps installed on them allow us to communicate more frequently & at greater distances than ever before. This capability is reshaping those communications. This centrality of the role of tech mirrors the importance that churches once had in peoples’ lives. The Internet Archive could be aiming to bring this back, in a more physical manner. Profiled in Tim Maly’s essay on Medium, the Internet Archive houses their primary archive and many of its servers in a church. As mentioned by Maly about digital librarian and founder of the Internet Archive, Brewster Kahle,

He wants the Internet Archive to be beloved. He wants closing it down to be politically expensive. So the virtual library is in a physical gathering place— they’re talking about opening up a café.

For this reason, he chose to house it in a beautiful building: a former church. There is something especially interesting in that choice to me, as well. Not only have churches (and other religious buildings) historically been gathering places for people, but they demand a certain sort of reverence. A historic gothic church is breathtakingly constructed with tons of stone. Perhaps that makes it all the more incongruous and intriguing to recognize such a stark collision of old and new–when digital technology and churches collide. In my travels I’ve visited many churches, one of them being the Votivkirche (Votive Church) in Vienna Austria. I took this photo in June of 2011:

The juxtaposition of the church lighting with the default screensaver of the iMac is one that doesn’t often reveal itself, and one that occurs in the church that houses the Internet Archive as well:

You’ll find yourself standing in a beautiful sanctuary. Ahead, the pews are still in place, facing the stage. Behind, looms the primary copy of the Internet Archive — three stacks of black servers nestled in an alcove and covered in blinking LEDs

The archive, in occupying a formerly sacred meeting space, embodies the social of the Internet. The hope to add a coffeeshop, too, rounds out the historical gathering places.

By making the Internet a literal institution, accessible by all and the meshing of old and new, of physical and digital, can continue. The Internet Archive stores a selection of the internet, as does the Barcelona Supercomputing Center. Both do so in former churches. According to Atlas Obscura:

What was once a hallowed place of worship known as the Torre Girona is now the home of the Barcelona Supercomputing Center, filling the expansive main hall with banks of futuristic computer equipment under glass.

The internet is made real in places all over the world, but only some create the dissonance of a community meeting place made real. A supercomputing center is perhaps a modern equivalent of a “meeting of the minds,” discussing problems facing the world and looking for solutions. Seeking answers in a place of worship is a bit different from seeking answers through supercomputing-powered analysis, but such are the intersections of meaning as structures are repurposed. Others have explored this modern dichotomy further, such as in Ian Bogost’s essay Cathedrals of Computation, or an essay by Michael Schulson on the intersection of religion and artificial intelligence. The shifting modes of community interaction and repurposing of buildings is also evident in the number of music venues that are converted historic movie theaters. The incongruity is evident in the stepped floor layout and the beautiful murals and detail on the ceilings and around the stages—now replete with lighting rigs and speakers.

The Politics of Crisis Communication

Sat, 10 Dec 2016 16:52:03 +0000

Some mentions in Zeynep Tufekci’s post The Politics of Empathy and the Politics of Technology got me thinking about crisis communications for incident response.

Facebook will have to decide which incidents are “serious and tragic” versus which ones are “ongoing crises” where Safety Check would not be useful. Iraq is not officially at war, but suicide bombings there are almost horrifically routine. Their new policy raises many important questions that should be carefully considered. Will Baghdad bombings be considered endemic? How many in a year to declare something endemic or chronic? Are we just acknowledging that people in the regions of the world suffering from chronic crises have no way to feel “safe”? Who gets to check in? “Useful” as defined as useful to whom? Would you not want a “Safety Check” everyday if your loved one were trapped in a region with a dangerous and fast-moving epidemic like Ebola?

When your platform takes the role of defining a crisis, that’s putting the media in social media. By enabling Safety Check for “unnatural” disasters, it stops being a nifty feature and becomes an essential tool for communication during a crisis.

Activating Safety Check constantly would lessen its value as a signal. Right now, it functions as a forceful push, and you get a “notify” on your phone when a friend in the affected area checks in as safe. Getting hundreds of these notifications per day would reduce its efficacy. However, not getting the notification when you were worried about someone would also be a problem. This type of system requires decisions to be made about when to activate, and when to hold back.

Avoiding alert fatigue is key when it comes to crisis communications. Facebook is both defining the crisis and communicating about it. It’s easy to send a message to millions of people when they log into your service on a daily (or even hourly) basis. It isn’t easy to do it well.

tweetthedocs: Use Twitter to meet your users where they are

Tue, 31 May 2016 13:30:50 +0000

As a tech writer, it’s hard to tell how users get to your docs at all. They might be clicking on in-product help links, searching the web, or getting sent links from support. But you can get proactive about it too. Help users of your product get their questions answered by meeting them where they are—on social media sites like Twitter. You may already rely on marketing, sales, support, and search engines to bring users to your documentation, but social media is a direct option. You can tweet about anything from general topics that answer common user questions to drier topics that are important for people to know. Read on to learn how!

Why to try tweeting your documentation

Using a content marketing approach like tweeting can help you reach users with your docs—and thinking from the perspective of tweetable product documentation can help you write more approachable, plain language and even improve the topic headings that you may be using.

Short sentences are easier for people to understand, but they aren’t always easy to write. You may end up explaining things in long, complex sentences because you’re still trying to grasp the topic yourself. If you struggle to come up with pointers or teasers to your docs, you may want to rewrite them to keep readers reading.

Restructure your sentences by thinking about how you might phrase a lead-in from Twitter. One-liners that describe a problem (or part of one) help people understand what they’ll get if they click—or keep reading.

“Here’s what’s new in this release” vs “Read this before you upgrade”
“About this dashboard” vs “Detect suspicious activity with this dashboard”
“Using this product” vs “Improve your security posture with this product”

Collect tweetable candidate docs

Google helps your users if they know what to look for, but Twitter can help if they don’t know where to start. Good topics to highlight on Twitter are those that answer common user questions, topics that may be dry but are important for people to know, tips and tricks that may be buried in a longer topic, and getting started information.

Examine the topics that aren’t getting much attention using site analytics. After you assess the writing quality and the location of each topic, determine if the information is useful (if it isn’t, that could explain the lack of hits). If the topic has a small, but essential use case, it could be that people who need it might not know it’s there.
Examine the topics that are getting a lot of attention using site analytics. Just like the lonely topics, the popular topics are great candidates for sharing. Docs with a lot of hits tend to contain vital information—so the more people that see them, the better.
Discover common struggles and use cases for your product with help from support. Your support team is your best ally—talk to them to see what customers ask about often, and what they’re trying to do with your product or service. You can answer those questions and address those goals proactively by sending your docs into the twittersphere.
Unearth hidden tips and tricks. Small paragraphs can hide important knowledge relevant to specific use cases. If someone is reading a topic for a specific goal, they may miss information that is good-to-know but not relevant in the moment. Highlight those bits of information with a tweet.

Plan your tweets

Once you collect a set of topics that are good to share, it’s time to get ready to start tweeting. If you have a marketing or communications team, work with them. They’ll help you avoid common pitfalls of social media marketing and communication, set up a content calendar, and define a voice that works with the company’s goals. Many of the steps for tweet planning are the same that you already use for doc planning.

Get an account. If you don’t have a company Twitter account to use, start one. Get permission to the main company one, or start a documentation-specific Twitter.
Schedule your tweets. Use a service that lets you schedule tweets. This lets you stay focused on your work, and tweet strategically instead of sporadically.
Know when to start. Pay attention to important times in your industry. A company-sponsored conference is a great time to start tweeting—your biggest fans will be there and ready to learn and spread the word after the conference is over. If your company isn’t large enough to have its own conference, glom on to another big industry conference, or just start tweeting!
Find your voice. Voice is important in writing, and even more so on Twitter. Treat the tweets like your docs—don’t patronize and don’t make assumptions. Evoke professionalism and trustworthiness. Be careful to use a voice that will resonate with the social media users—don’t sound too stilted, too corporate, but also not too friendly.
Define your audience. Determine which portion of your customer base is likely to be on Twitter. See what your competitors are doing.
Plan your frequency. If you’re joining an existing Twitter account, pay attention to how often that account tweets so that you don’t overwhelm the existing content. Post often enough to make following the feed valuable to readers.
Time it right. Decide what time of day your tweets should go out. If your audience is all in one location, pick times when you think they’ll be online. If you have a global audience, keep that in mind and plan to target specific segments of your audience at specific times of day.

Start tweeting

Start writing your tweets!

Write a backlog, and plan meetings in the future to refill it. This is the kind of backlog you want to have—a backlog of content to put on the internet. Write up 20-30 tweets before you start tweeting to give you a good start. Schedule meetings in the future with yourself and anyone else helping you out to write new ones—checking the analytics to see if specific times, verbiage, or topics resonated with your audience, and try new ideas too!
Include graphics to encourage engagement. Include diagrams and screenshots to help your tweets stand out. Don’t include them in every tweet, and not just for the sake of it, but rather when they’re useful.

After you start tweeting

Pay attention to what’s happening on Twitter.

Get feedback. Have regular meetings with the people who manage the Twitter account (if it isn’t you) to hear about how your tweets are doing, and learn what topics to prioritize in the future
Watch the feed. Be cautious around events or news that dominate social media posts for awhile. It’s obvious when you’re using a scheduling service if your tweets about how to use your product show up at the same time everyone is livetweeting a political debate or reacting to a tragedy in the news. On the other hand, stay attentive to trends that may be relevant to your product. If you make a security product, a security vulnerability or breach may be a good time to tweet more often.
Monitor the feed. Make sure someone is dedicated to monitoring the feed and addressing replies from customers and potential customers (and the spambots and bullies).

Tweeting the docs

Improve your documentation and collaborate with other teams to #tweetthedocs. Write better sentences and headers, without sounding too much like a marketing #brand that you alienate your readers. Don’t wait for Google to bring customers to your docs—reach out to them proactively!

Where Scanning the Internet Gets You

Wed, 20 Jan 2016 19:25:45 +0000

From awhile back, Brian Krebs talks to three researchers at U-M about their ZMap tool. An efficient and comprehensive way to scan the Internet, they’ve recently built a search engine called Censys that searches across their daily data collections from the ZMap scans. From Krebs’ interview with the researchers (Zakir Durumeric, Eric Wustrow, and J. Alex Halderman):

“What we were able to find was by taking the data from these scans and actually doing vulnerability notifications to everybody, we were able to increase patching for the Heartbleed bug by 50 percent. So there was an interesting kind of surprise there, not what you learn from looking at the data, but in terms of what actions do you take from that analysis? And that’s something we’re incredibly interested in: Which is how can we spur progress within the community to improve security, whether that be through vulnerability notification, or helping with configurations.”

Using ZMap allows them to quickly collect this data (compared to other network scanners), but the researchers aren’t just scanning the Internet because they feel like it. They’re taking action based on the scan results—notifying people when their machines are vulnerable to the Heartbleed bug.

Beyond notification, they can take other steps:

“So, that’s the other thing that’s really exciting about this data. Notification is one thing, but the other is we’ve been building models that are predictive of organizational behavior. So, if you can watch, for example, how an organization runs their Web server, how they respond to certificate revocation, or how fast they patch — that actually tells you something about the security posture of the organization, and you can start to build models of risk profiles of those organizations. It moves away from this sort of patch-and-break or patch-and-pray game we’ve been playing. So, that’s the other thing we’ve been starting to see, which is the potential for being more proactive about security.”

Internet scan data can help us better understand organizational security posture and develop different models of risk profiles in organizations. With those risk profiles, improving an organization’s security posture could be a matter of identifying the inefficient elements and focusing on them. Security posture is culture as much as machines. While SIEMs can identify risk factors in your machines, models of organizational security posture can identify the risk factors in your culture.

Kill Legacy Apple Software

Mon, 11 Jan 2016 20:30:14 +0000

Benedict Evans pointed out in a recent newsletter, “there’s a story to be written about Apple feeling its way from a piecemeal legacy technology stack for services, evolved bit by bit from the old iPod music store of a decade ago, to an actual new unified platform, something that it is apparently building.”

I’d argue for a focused set of decoupled applications, rather than a new unified platform. iTunes has bloated beyond practicality. The App store doesn’t work well for users or developers. Here’s where I think the future of these applications lies.

iTunes

Focus on the music. Start over and build a new app that focuses solely on music. Ensure that it supports previous iTunes metadata, either as an import or reading iTunes library files directly.
Improve smart recommendation features. Genius works well, when it works. It only works on Apple-imported or downloaded songs, which isn’t many.
Improve cloud streaming. Apple Music is probably the answer to this, but I haven’t tried it. iTunes Match is another service that offers something similar, but with similarly limited adoption. Integrate cloud streaming offerings, and do a better job making it clear to the customer what they’re getting when they sign up for these services.
Refocus the UX on music-based workflows, like playlist-making and music listening and exploring.
Enable better playlist management:
- Playlist archiving to save mixes that you’ve shared with someone, but don’t use in your day-to-day interaction with the app.
- Playlist freezing to preserve the order of a playlist so that a perfectly designed playlist can stay that way until you unfreeze it.
Maintain a Radio section. Not only that, but store and highlight downloaded radio streams in that section. That way a livestream won’t come up in shuffle, but I can still keep my favorite college radio station streams close at hand.
Improve metadata management. iTunes can’t compete with high-fidelity-focused services when it does such a poor job of managing file metadata.

App and Media Store

Drive purchases through one central store. I’m terrible at naming so won’t propose a name. Allow purchase of applications, media (movies, music videos, etc.), and music.
Position the store as a platform. Allow the new music app to integrate on a stream-to-buy interface smoothly. Let other developers more smoothly call up their app page in the store.
Offer app recommendations, on an opt-in basis. The App store knows what apps I have installed on my phone. If I want, I can have it recommend new apps to me to try. Recommendations can be improved even if they aren’t personalized.
Improve findability and search. I shouldn’t have to know the precise name of an app to reliably find it. Improve the search algorithms.
Improve whatever developer interface exists. Offer more insight into the app approval process and the reasons for an app rejection. Treat OSX developers equitably with iOS developers. Improve the developer-customer interface.

Device Manager

Build a new app exclusively for device management. Using iTunes to manage an iPhone is a legacy embarrassment and confusing for new phone users. Liberate the device management function from the music app.
Improve content management. Expand the current device management interface to allow you to:
- Add or remove apps, potentially with an app store interface.
- Add or remove music, either with a file-based interface or a link to the new music app.
- Add or remove photos, with a file-based interface or a link to the photos app.
- There are security concerns to enabling purely file-based content management, but it’s worth considering.
Handle backups and updates smoothly.
Interface with the store to buy apps. If I choose, I can then download new apps for my device from my computer.

Podcasts

Give podcasts their own app. Highlight them in a tailored app. It could use the iTunes platform without being a part of iTunes. Having a standalone podcasts app would consider the iOS and OSX convergence that Apple seems to have begun by introducing Photos.
Podcasts use a subscription model, with different metadata and descriptions than is valuable for a music listener. For those reasons, they don’t belong in an iTunes focused on music.
Consider introducing a sharing element. Copyright and intellectual property would likely prevent snippet sharing of many podcasts, but it seems like it would add value.

Why is this so important? First of all, I’m dedicated to iTunes, yet I still hate it. Everyone has gripes about these common software programs, yet we can’t stop using them because we must. As Micah Singleton points out for The Verge:

“If you’re like me and have avoided iTunes save for the occasional Adele purchase every four years, you don’t want to go back to using a bloated program when Spotify’s app is fine, Pandora works on basically every device, and YouTube is where you spend most of your day avoiding work anyway.”

Singleton agrees that iTunes could be much improved, and adds the additional suggestion that Apple should iterate faster:

“Apple can’t wait until WWDC every June to make significant improvements to Apple Music. It should move to a more frequent update schedule to keep people engaged in the service.”

If, according to the grand unified theory of Apple products, “Apple’s fundamental goal [is to] make technology more personal,” it needs to ensure that its software is doing that as well as its hardware and operating system improvements. If “personal technology is about taking complicated tasks and breaking them down into more granular tasks,” then it follows that Apple’s software should be trying to do the same thing.

One-stop-shop behemoths of software should be a thing of the past. Instead, focus on building a platform with practical apps that interact with its framework. Our devices are the one-stop-shop, and our apps should fit specific needs rather than attempt to do everything.

Ward off the bloat, divest app functions, and bring back the love of Apple software.

On Broken Music Discovery and the Fragmentation of Music Libraries

Wed, 02 Dec 2015 20:25:15 +0000

Following up (finally) on a tweet-storm from March about music discovery and libraries now.

I miss old school mp3 blogs. Music discovery feels broken now that it’s fragmented across so many services. Soundcloud, Spotify, iTunes… — Sarah K Moir (@smorewithface) March 23, 2015

I used to subscribe to lots of MP3 blogs. I had lots of free time in high school, and listened fervently to the local college radio station (as I’ve mentioned before, in an autobiography through musical devices.) Music discovery is now fragmented across services—SoundCloud, Spotify, iTunes, Pandora, the now-defunct Rdio, and even 8tracks)—it’s both harder and easier to find new music. The wizardry of Shazam, too, means getting to find out what song is playing in the bar, store, or on the radio so you can buy it or find it later online.

Wish there was still a “my library” out there that would actually be mine. Google play music tries, but login is awkward. And interface too. — Sarah K Moir (@smorewithface) March 23, 2015

I still use iTunes, but it’s a bloated program now, without a clear goal or identity as a software application. It started as a music/media player to compete with Windows Media Player (as someone who used my first iPod on a PC, that’s the way it seemed). Now it’s the one-stop-shop interface with the iTunes store (though not the app store), music library management, movies, podcasts, radio, AND the interface for updating & backing up i devices.

It got harder to identify with my library as mine, since the software holding it is a beast of processor-hugging resources and multiple discordant functions. As mp3 blogs declined in popularity after RIAA crackdowns and mass shutdowns, my music listening and storage habits fragmented. I turned to SoundCloud and Grooveshark to listen to new music discovered on blogs and by friends.

These days my library is all over the place. iTunes is still the main home of my music—I can afford to buy new music when I want —but I frequent Spotify and SoundCloud to check out new music. I sync my iTunes library to Google Play Music too, so I can listen to it at work.

Google Play Music is in the cloud, but doesn’t handle the transition from my iTunes library in a seamless way. Playlists aren’t synced over from iTunes, and the 1-5-star ratings that I use to organize my library are reduced to an overly simplistic thumbs-up or thumbs-down rating, automatically and opaquely updated at music upload time.

Using all these different services is putting a bandaid on the main issue I find.

Hard to feel as connected to music anymore when it lives in so many different places, and recommendations are harder to find… — Sarah K Moir (@smorewithface) March 23, 2015

It’s harder than it used to be to feel connected with music. It’s not a stream or a subculture one is tapped into anymore, because it’s so distributed on the web. There’s so much music, and it lives in so many different services, that the music culture has imploded a bit. Music isn’t alone in this regard; it’s also affected television, courtesy of Netflix:

“We can’t talk about buzzy Netflix shows because our schedules are out of sync. The rough expectations for knowing if your friends are on episode 12 or episode 1 have been destroyed. Netflix thinks it has performed a noble act by releasing the entire season en masse, but it has actually wreaked havoc on the best part of television: talking about television.”

But I think I was a bit wrong in March to say that recommendations are harder to find. Good recommendations are harder to find. Recommendations in general, are not. Spotify launched the Discover Weekly playlist to try to algorithmically guide someone through new music once a week. An opaque algorithm without a way to give feedback beyond play count and a rating isn’t ideal, though.

When I was in high school and college and defining my music taste, it was important to be the first to discover the “next big thing” in music. There was an element of superiority to this too, extending to the hipster put-down, “Oh, I’ve been listening to them for ages” when you tell someone about a band you just discovered.

Part of breaking down that cultural elitism has led to a pipeline fracture. You could find recommendations easily in trustworthy music blogs, but many of them have shut down. The launch of the Hype Machine both helped and hurt the cause—not only is it easier to find the next big thing before most others know about it, you’re all riding the wave together. But music blogs themselves get less individual traffic, and the number of songs that get hyped on HypeM are intimidating to sit down and listen to and choose your favorites.

And take more time to parse through. Or maybe that’s just the post-college desiccation of RSS feeds and my time. — Sarah K Moir (@smorewithface) March 23, 2015

HypeM only captures a sliver of the music trending today. People have their favorite bands that they’ve discovered on services like Bandcamp and SoundCloud that let artists upload their music directly, or others like 8tracks or Rdio where you can create custom stations or playlists to share on the web, or relying on the algorithmically generated recommendations of Spotify Discover and Discover Weekly, SoundCloud’s recommended tracks, and so many others. And music radio has consolidated to a few pop stations that play the same songs, one alternative/indie station if you’re lucky, and maybe a rap/hip-hop station or two.

For the first time in my life I’ve been listening to top 40 radio. I still love local radio too. A lot of it still feels one-noted though. — Sarah K Moir (@smorewithface) March 23, 2015

Still the same old songs over and over again, but different genres. — Sarah K Moir (@smorewithface) March 23, 2015

Radio may always be like this, and everything overplayed to death. There are so many places to find new music, and my time is drying up. It takes a lot of dedication to sort through the latest in new music releases, and that’s only become more difficult as the number of sites and sources of new music has expanded. You can’t dump a bunch of blogs into your bookmarks or an RSS feed to discover the latest hotness. Now you’d need to check Spotify, Bandcamp, 8tracks, SoundCloud and others with their own embedded audio tracks to see what’s new on the music web.

I’ve been a long, long time music listener. From wandering the library, to working in college radio. And I still can’t figure out a solution — Sarah K Moir (@smorewithface) March 23, 2015

Discovery of music and sounds across the Internet has come up time and again with podcasting and audio on the web. While it’s relatively easy to distribute music on the web, the community listening to the music is fragmenting. Sites like Last.FM used to be a music community, but now merely passively collect the scrobbles (listens) of its users. As the venues of music listening have fractured, so too have the communities alongside them.

Algorithms can only do so much. Spotify’s discover is broken. Soundcloud’s recommended tracks is helpful, but works on track-by-track basis — Sarah K Moir (@smorewithface) March 23, 2015

I find it silly that SoundCloud, and even YouTube’s autoplay feature work almost exclusively on a track-by-track basis. Maybe this indicates how far machine learning still needs to come. Siri can tell me what I’m listening to, but can’t answer any follow-up questions about the music. Even Spotify’s Discover Weekly playlist is still learning that a person’s music listening habits are greater than just one song (or what they listened to most recently).

People no longer want to develop Pandora stations just around one song, or one artist, but a mood or several sounds. Google Play Music and Spotify attempt to address that with their radio and playlist options. Songs for “focusing” or “retro soul” allow you to listen to music more idly. You’re discovering new music without needing to try. You could listen to new music all day and by the end of the day not know what you listened to. You can collect songs by clicking a + to add it to your library, without having to buy it.

It gets harder & harder to take radio songs & put them in my library. Maybe that’s what it means for the digital age to transform ownership. — Sarah K Moir (@smorewithface) March 23, 2015

I can ask Siri what I’m listening to, and click buy right on the response screen, but I never do. Owning a song won’t fix the fracturing of my music library. I keep track of the services I use to listen to music, and what bands I follow on them, in an effort to stay up-to-date on the new hotness. The alternative is to buy everything that I want to listen to multiple times.

Even if I did buy all the songs I want to hear over and over again, I’m not sure it’d fix it. I own most of them, but it still feels off. — Sarah K Moir (@smorewithface) March 23, 2015

The device I carry with me everywhere isn’t big enough for my library. Even when it was, it still wouldn’t be my go-to vs the Internet. — Sarah K Moir (@smorewithface) March 23, 2015

Ownership doesn’t automatically translate to access. Just because I own something, digitally, doesn’t mean it’s “easy” to keep close at hand. My CDs are all in one place. I can look at them on the shelf. They can only be played in one type of device, which is only easily accessible in my car (RIP Macbook optical drive). Digital music, even when I own it, sits on my computer’s hard drive and needs to be transferred before I can listen to it when I don’t have my computer.

My iPhone isn’t large enough to hold any of my music library. My iPod is small, ancient, and doesn’t hold a charge. It does the job for my daily commute on the train, but it’s not ideal. Mostly I use Google Play Music, to listen to my library when I’m not at home, but I need an internet connection (or a subscription) to do that. And again, a fragmented library means that I can’t access songs exclusively in Spotify or SoundCloud on the go, or at home without having multiple programs and websites open.

Though so much of the web revolves around sharing now, it’s still a challenge to share music with others. Mixtapes and mix CDs are a thing of the past, and sending songs back and forth on Spotify is the closest you usually get to sharing your music taste with others. Although any kind of music-sharing infrastructure is immature and soon-to-be tangled in future RIAA lawsuits, that likely isn’t the only reason that music sharing has declined for me. The time I devote to listening to new music has shrunk. Probably in line with the difficulty of tracking it down myself, but also as I’ve gotten busier. Attention spans have shrunk, and music is constructed to take advantage of that. The Swedish takeover of pop songwriting means that:

The producers compose the chord progressions, program the beats, and arrange the “synths,” or computer-made instrumental sounds; the top-liners come up with primary melodies, lyrics, and the all-important hooks, the ear-friendly musical phrases that lock you into the song. “It’s not enough to have one hook anymore,” Jay Brown, the president of Roc Nation, and Dean’s manager, told me recently. “You’ve got to have a hook in the intro, a hook in the pre-chorus, a hook in the chorus, and a hook in the bridge.” The reason, he explained, is that “people on average give a song seven seconds on the radio before they change the channel, and you got to hook them.”

The music listening experience is designed for us by a bunch of Swedes, writing songs and producing countless hits (Max Martin and his ilk), developing a streaming app that has taken the industry by storm (Spotify) and a website that filled the gap left behind by mp3 blogs (SoundCloud). While I don’t have a solution for the fragmentation of my music library that I’m grappling with, maybe someone in Sweden will.

Security Communications: Lessons from the National Weather Service

Thu, 12 Nov 2015 17:33:14 +0000

Security communicators can learn some lessons from national disaster communication. Motherboard interviewed Eli Jacks, chief of the National Weather Service’s Fire and Public Weather Services branch in The National Weather Service Wants You to Be Scared of This Blizzard, and Jacks shared many important elements of their communication strategy.

“We want people to notice and take action.”

This is a key goal of the NWS, and clear action items are necessary to fulfill this goal.

“the average person doesn’t always know how to interpret the warnings and think it’s like some of our other ones.”

Communicating the appropriate level of risk is a key challenge, because it helps people decide whether taking action is a high priority for them.

“Overhyping this potential storm and having it underwhelm is less of a risk than not adequately warning people.”

The NWS also must make this intentional choice when communicating because storms can cause death, injury, and massive power outages. It is better for people to overprepare and be unaffected than underprepare and die.

“It’s almost less important that we actually get the call perfectly right than the public responds to it and perceives it as being dangerous.”

Alongside overhyping a storm, the NWS chooses to focus on getting a message out rather than making it perfect. Storms are time-sensitive, and it is often most important to get the message out, and fast.

“better targeting our messaging so that we’re reserving dire warnings for when they’re really going to occur.”

Since weather is all about forecasting, this is perhaps a natural future goal of the NWS. Overhyping a storm is okay once in awhile, but accurate predictions are vital to get people to trust and consistently take action when they should. To distill these lessons, there are three main challenges when communicating threats, whether they are security vulnerabilities that threaten cyber security or weather patterns that threaten people’s lives.

Get people’s attention. It’s important to reach as many as possible who might be threatened by a weather pattern, just like the first step in helping people stay safe in the face of a security vulnerability is getting the attention of those who are affected and need to take action. The urgency of a message (if the weather system is growing in power, or if criminals are actively exploiting a severe vulnerability) is key to assess when your message needs to reach a lot of people.
Write accurate and easy-to-understand calls to action.
- Use clear language to accurately express the severity of the situation. The more accurate your communications, the more you will be seen as a trustworthy and reliable source.
- Avoid histrionics when writing calls to action, but don’t be too conservative when describing the risk. Help people understand the severity of the situation by including risk factors. For example, you would be more at risk during a hurricane if you live near the coast or in a flood-prone region. Similarly, you would be more at risk for heartbleed if you reuse passwords across accounts and websites.
- Make it clear what people can do to protect themselves and prepare for future threats.
Avoid alert fatigue. If you notify people too often about dangerous weather patterns (or security vulnerabilities) that in the end, don’t turn out to have much of a noticeable effect, you may lose the trust of your audience, or they may lose the energy to react properly in the face of constant security (or weather) threats.

The first challenge, attention-getting, helps explain why so many serious security vulnerabilities are communicated to the public with catchy names, dedicated websites, and even logos. Names, websites, and logos make it easier to get the attention of an affected audience and the media. As message targeting improves, this challenge may be simpler to address.

The second challenge is the most difficult to address. Security communications exist solely to drive remediation of security vulnerabilities. Whether people must patch systems, take them offline, or implement a workaround, that information must be shared as quickly, accurately, widely, and simply as possible. Not easy to do.

Higher-profile security vulnerabilities also help provide a risk gauge for future vulnerabilities, easing the effect of the third challenge, alert fatigue. While the quantity of alerts is still important to monitor and restrain when possible, providing a risk factor (even a relative one based on higher-profile vulnerabilities) makes it easier to process the severity of a security issue.

Names for security vulnerabilities are valuable when addressing the three primary security communication challenges above, but naming carries risks that you must be aware of when communicating.

Naming something gives it power. A name is a lot easier to track than an archaic set of technical terms or a string of letters and numbers. For example, the FREAK attack, has both a scary (and thus memorable) name and is much easier to remember than “Factoring RSA Export Keys,” the more technical description of the vulnerability. CVE identifiers are necessary, but not easily to remember.
Names encourage media attention. This can lead to more attention than a vulnerability warrants, and overblown fear. Because of this, it’s important to gauge the severity and the impact of a security vulnerability before naming it. Any awareness is usually better than none when it comes to computer security, but avoiding alert fatigue is key.
Names must be intentionally chosen. What you choose to name something is important, because if the words in the name are frightening (like FREAK), they will provoke a different reaction than you might want or need. A name should prompt recognition and response, rather than fear and panic.

Whether you are the protecting the nation from severe weather patterns or alerting your community to security vulnerabilities in your product, the National Weather Service provides a good model for effective threat communication. It’s difficult, but possible with careful consideration of the challenges and risks involved.

Affective Computing and Adaptive Help

Wed, 14 Oct 2015 10:06:00 +0000

Several months ago, I saw Dr. Rosalind Picard give a talk on Affective Computing. I took notes and thought a lot about what she said but let my thoughts fester rather than follow up on them. Then last week, I read Emotional Design by Donald A. Norman, which reminded me of Dr. Picard’s work and my initial thoughts about affective computing. There are two elements to affective computing:

People interact with technology and devices as though it has a personality (and devices and interfaces without personalities can be distasteful to use).
Cameras, wearables, and other technology can be used to determine the emotions and affective responses of a person using technology with surprising accuracy.

Websites and applications are personalized by tracking your browsing history, collecting advertising preferences, device usage, and demographic data. Using affective computing, they could soon be personalized by tracking your emotions.

Help tailored to you

The great challenge of online or in-product help is providing help you want, when you need it and just-in-time help, offering guidance before you get frustrated with the experience. This is commonly referred to as adaptive content or adaptive help. Using advances in affective computing, adaptive help could be customized not only based on your knowledge of a product, what device you’re using, and where you are located, but also based on the emotions you experience as you use the product. By identifying capability personas, it’s easier to understand what types of help certain types of users might look for, and what kind of help they might need.

Have you logged in and performed x number of actions over the last week, and each week you confidently perform at least one new action successfully? Congrats, you’re now a part of the “confident autonomy” group.
Have you logged in once a week and performed the same action every time, occasionally flitting to other views within the application but never completing anything but that one action? Congrats, you’re part of the “methodical novice” group.

As a member of the confident autonomy group, you might look for suggestions about what types of actions to learn next, while as a methodical novice, you might want to know about process improvements or how to expand your use of the tool. Knowing when to introduce that help is also key, and this is where the inclusion of affective computing could be vital. You might want help right when they start to get frustrated, indicated by your forehead wrinkling in irritation. Or perhaps if you click through the beginning of a process repeatedly, but never complete it, looking confused all the while. Successfully addressing these scenarios is challenging. It’s hard to understand and anticipate the “why” behind someone’s emotions and actions. We can’t know what is frustrating someone. We can’t always find out what is stopping someone from completing a process.

A second iteration of Clippy?

Microsoft tried to anticipate the “why” behind people’s actions in a rudimentary and now-notorious implementation named “Clippy,” designed to respond when it noticed certain patterns in the Microsoft Office. Start a paragraph with “Dear Susan,” and Clippy would appear and ask if you were writing a letter, and if so, could it be of any help? The capability now exists to make a Clippy-like tool far smarter using machine learning and affective computing. This tool could take in a large amount of data before making suggestions.

No longer as simple as recognizing a structure of “Dear Susan,” the tool could make note of how many times an account has written Dear Susan before. For people who write letters often, the tool could “learn” that they probably don’t need assistance writing a letter. For someone who has never written “Dear Susan” before, the tool could take that into account before offering advice.

Using affective computing, the tool could monitor your facial expressions, or sync up with a wearable device, to assess your emotions as you engage with a product. If you’re bored, excited, or confused, Bindy could offer varying suggestions. Truly adaptive help could be responsive to each of these signals, changing the help message accordingly. Rather than ask if you needs help writing the letter, the tool could ask instead if you need help writing the letter faster, by pointing to some templates. the tool could also suggest help for sending a letter to multiple people with mail merge tips. If the name matches someone in the user’s contact lists, the tool could automatically suggest (a la Crystal) ways to better target their audience.

Surveilling you to help you

The obvious challenge in this day and age is that the technology required to make this work requires near-pervasive tracking—a usually unwelcome act of surveillance. Software to observe and monitor your emotions, actions that you perform in a product and on a device, that integrates with your contacts and friends lists would require you to place a lot of trust in a product or service and its ability to protect your privacy.

However, this isn’t too far off from what currently exists in many of our devices. As I draft this in Google Docs, my writing history is tracked. Google knows exactly how many times I have (or have not) written “Dear Susan” and similar formulations. Google also has access to my contact lists, data on how often I contact them, and a convenient way to set up a video feed of my face integrated in Google Hangouts and Google Mail. The new element of the tracking you to provide you with a personalized experience is affective computing. Currently, the most effective form of affective computing requires a webcam or another way to track your facial expressions. Other types of affective computing rely on voice-based analysis. But as wearable devices become more common, and integrate heart rate monitors and the movement of your wrists as you type, the potential for the integration of affective computing into a daily workflow becomes more realistic and subtle than a constant video feed of your face.

Adaptive help won’t stop butting in

In addition to the necessary surveillance, the biggest challenge to this entire concept is a challenge the Nielsen Norman Group identifies in all adaptive help and the reason why the original Clippy failed: “it appeared without the users’ direct request. Like pop-up windows, the animation surprised and annoyed users who hadn’t wanted help”. Clippy, or any potential replacement tool, would be sharing potentially helpful information in the least-receptive manner possible. Unasked-for, with no understanding of your thought processes, much like your parents when you were a teenager. Another risk is also one that strikes parents of teenagers, as the Nielsen Norman Group makes clear: ”If the system suggests topics that users aren’t interested in, they will quickly learn to disregard those suggestions.” These adaptive help systems would have to be finely attuned to their user’s help needs, and provide that information at just the right moment (or determine whether to provide it at all).

Challenges of “smarter” adaptive help

Scrunching up your face isn’t a request for help—it could be a sneeze (it’s allergy season) or it could be the frustration right before you make a triumphant leap in your knowledge and understanding. Even the best-designed adaptive help system, using machine learning and affective computing, faces a near-insurmountable challenge. As Donald A. Norman reminds us in Emotional Design, “The proper response to an emotion clearly depends upon the situation.” The proper response to frustration isn’t always immediate help, and understanding the cause of the frustration is essential. A few examples from Norman:

“If a student is frustrated because the information provided is not clear or intelligible, then knowing about the frustration is important to the instructor [in this case, the adaptive help interface/intelligence], who presumably can correct the problem through further explanation.”

Incorporating an adaptive help response at this point would also not fix the true issue. If a person has gotten far enough only to be frustrated by unintelligible information, there is likely a user experience or interface issue at fault. Offering help is not the solution for every problem.

“If the frustration is due to the complexity of the problem, then the proper response of a teacher might be to do nothing. It is normal and proper for students to become frustrated when attempting to solve problems slightly beyond their ability, or to do something that as never been done before.”

Providing a helping hand here could, in some cases, stunt your growth as you become more comfortable with a software product, as you begin to rely not on your own experience with the product but instead lean on the adaptive help to tell you what to do. Norman continues, “if students aren’t occasionally frustrated, it probably is a bad thing—it means they aren’t taking enough risks, they aren’t pushing themselves sufficiently.” In the case of people that fit the proficient novice capability persona, this could be an opportunity for the adaptive help to push a person toward new ideas, concepts, or functionality that they may not have tried before. Another point of Norman’s is relevant for this specific frustrated student too: “It probably is good to reassure frustrated students, to explain that some amount of frustration is appropriate and even necessary.” Not all frustration is a problem to be fixed. When providing adaptive help, understanding the cause of frustrations is essential. Affective computing could help here.

“if it goes on too long, however, the frustration can lead students to give up, to decide that the problem is above their ability. Here is where it is necessary to offer advice, tutorial explanations, or other guidance.”

Using available data to help understand when a user might want help is the best use case for adaptive help. Norman’s explanations of the different types of appropriate reactions to frustration offer many cases where it may be inappropriate to offer help. The appearance of our Clippy replacement tool, whether it takes the form of a cleverly designed pop-up or slide-in modal, will be summarily rejected, along with its suggestions, if it appears at an inappropriate time. In contrast with many of the software development goals in the industry, adaptive help doesn’t leave much room for iteration. User trust is not easily won back once lost.

Give adaptive help feelings

To get it right the first time, we need to give adaptive help some feelings. An essential element of emotional design, identified by Norman in Emotional Design, is that the system has to have its own emotions as well, and the right kind for the situation. Style guides emphasize voice and tone for writing, and the same is true for adaptive help. Our tool would need a personality. Clippy made us feel stupid and wasn’t very smart itself, yet it was easy to personify—it even had eyes and a mouth. An ideal replacement tool would have a personality, necessarily targeted at the emotions of its users. Systems built without emotions are often nevertheless personified, so it is better to design for the inevitability than to design an emotionless help-providing system that feels cold to its users.

Let users ask for help

If you could predict when Clippy’s replacement would appear, or know how to reliably summon it when you knew you needed help, the tool could be more helpful. Chat robots already allow you to reach out and ask a question, though their subsequent helpfulness is debatable. You can summon slackbot in chat with a keyword or phrase, or ask it a question directly. Occasionally, the help menu in a software application lets you ask for help. More often, asking for help in the help menu sends you to a forum, the homepage of the documentation site, or a list of not-quite-relevant yet “frequently” asked questions. Making the appearance of Clippy’s replacement predictable or summonable makes it a friendlier help experience. Less of an unpredictable black box—say, someone that shows up and walks into your house when they want to—and more of a good friend that shows up when you call them, but also sometimes just when you need them, whether you called them or not.

Building a replacement Clippy

Adaptive help to this degree seems almost untenable. To build a help system this responsive and involved, you’d probably be better off investing the money in more customer support staff, designers, and technical writers. Building this type of tool would be technologically involved. You’d need several different elements:

Machine learning, to better characterize the habits and patterns of application users. Define who is confidently autonomous, who is a methodical novice, and who is an outlier.
Affective computing, to know to offer help when someone is frustrated, confused, or irritated with the application, and to understand how to frame the message. A frustrated person probably needs to hear a different message than a confused person.
Emotional design, to lend a friendly, understanding character to the help content.
Anticipatory computing, to take machine learning a step further and also allow a way to correct the help suggestions.

Building the actual help content that would power Clippy’s replacement would be even more challenging. Rather than writing content for the most common denominator, we would instead be writing for every possible denominator. While challenging, models exist for building adaptive content. And if done properly, adaptive content allows us to properly control the context that our content appears in. When we know the context, we can write properly understanding content.

The present state

An adaptive help tool like a replacement Clippy could be valuable, but simply not worth the investment. The risk of losing a person’s trust in an application or website, coupled with the complexity, surveillance, and gray areas involved in building such a tool make it impractical. However, each element that I outlined to structure a replacement Clippy exist in various application contexts. Machine learning is being used to drive anticipatory computing to tailor messages and notifications to you when you need them using data like your location, device, and how you use an application. For more on anticipatory computing, see How A Computer Can Anticipate Users’ Needs (Without Driving Them Crazy) in Fast Company, by Paul Montoy-Wilson. Kristen V. Brown sheds some light on other applications of machine learning in her piece for Fusion, I tried three apps that claim to make you more likable—and am now addicted to one of them.

Affective computing is primarily being used to gauge the effectiveness of advertising campaigns, ensuring that ads will continue to grow more targeted. See Raffi Khatchadourian’s piece in The New Yorker, We Know How You Feel. Andrew McStay discusses the wearable component of affective computing in The Conversation, Soon smartwatches will listen to your body to work out how you’re feeling, and much of Rosalind Picard’s groundbreaking work in affective computing involves wearables as well.

Nathan Collins discusses voice analysis and affective computing in his piece for Pacific Standard, You Sound Sad, Human. Emotional design has had an entire book devoted to it by Donald A. Norman. Beth Dean discusses Facebook’s efforts to incorporate Emotional Intelligence in Design on Medium, while Neil Savage discusses the personalities of robots in Artificial Emotions for Nautilus. Andrew Wilkinson compliments slackbot in his piece Slack’s $2.8 Billion Dollar Secret Sauce on Medium.

Adaptive content is discussed on Firehead’s blog by Noz Urbina in What is adaptive content? and Kate Sherwin assesses the design and user experience of adaptive help interfaces for Nielsen Norman Group in Pop-ups and Adaptive Help Get a Refresh. Especially important to consider from a design and content perspective are Sara Wachter-Boettcher’s words on Everybody Hurts: Content for Kindness.

Protect Your Accounts and Devices

Sun, 11 Oct 2015 17:45:24 +0000

In honor of #NCSAM, here are some tips for keeping your accounts and devices safe on the web.

Don’t reuse passwords. Better yet, use a password manager like LastPass. Why? If someone gets access to your username and password for one account, only that account is vulnerable. But if you reuse passwords, they now have access to multiple accounts. If someone can steal your Pinterest password and then log into your bank account, that’s a problem.
Set up two-factor authentication. This is when you use something in addition to a password to log into an account. This helps keep criminals out because they have to know your password and something else to get into your account. The “second factor” is typically a code that you get (from a device, an app, or a text) that you enter after you log in to a site with your username and password. The code is usually time-limited, so you have to enter it within a certain period of time, otherwise you can’t get in.
Password-protect your devices. If they get stolen, the criminal can’t access your personal information. Bonus: nosy people also can’t access your information.
Close accounts you don’t use. While harder to do and remember, if you’re not logging into an account often, you’re less likely to notice if it gets compromised. Often dormant accounts can have information in them that could be useful for gaining access to another of your more valuable accounts.
Keep your important email separate from your junk email. Do this in address at least, if not inbox. Use a separate email address for making throwaway accounts or signing up for newsletters, and another one for more valuable accounts. You can focus on protecting the more valuable account, and share the less valuable email address more widely. Keep linkages between the accounts low.

Tips for your devices.

Update software on your device as soon as possible. If you get a software update that contains security updates, install it as soon as possible. Most operating system updates will have security fixes. Turn automatic updating on when possible.
Use anti-virus. Still one of the best ways for you to protect yourself against common malware and adware. Keep this updated too.
Don’t click on stupid shit. If it looks suspicious, don’t click on it. If it seems too good to be true, don’t click on it. If a popup won’t let you close your browser until you click on it, definitely don’t click on it (force-quit your browser instead).
Use an adblocker or block Flash. Use uBlock Origin as an adblocker, or set Adobe Flash Player as click-to-play, or disable it if you don’t watch videos often. Disable or uninstall Shockwave Flash altogether. This protects you from malware distributed through advertising, also called malvertising.
Install trusted software. What does it mean to trust software? You’re downloading it from Apple’s App Store, or the Google Play store. Reviews are positive. If it’s a website, it’s the developer’s website or CNET. When you download it, you don’t have to spend five minutes trying to decide which download button is the safe download button.

There are many more tips that I could recommend, but as it is, I only follow half of these—and that’s why I’m posting them here. They are best practices that are hard to keep up with. So start with one, or two, and every few months (or every year, in honor of National Cyber Security Awareness Month) add a few more good habits to your online life.

Discomfort, Trust, and Digital Selves

Tue, 08 Sep 2015 23:19:54 +0000

It’s been awhile. I’ve spent the last four months applying for new jobs, interviewing, getting hired, and moving from the midwest to the bay area. It’s been a long ride (drive, really). I’ve been out here three weeks now, and it still feels strange to call it my new home (new license plates on my car notwithstanding).

I’m a tech writer by trade, as I’ve alluded to/mentioned in the past with my post on Prescriptive Design and the Decline of Manuals, and I’ve so far enjoyed being in an area so tech-focused (though I do worry about the bubble).

Let’s get back into it, shall we?

I read a book called The Antidote on the train last week. It was a good (and timely) reminder that big life changes like moving across the country are good for you. It’s refreshing, and makes you happier, if you can learn to revel in the discomfort of the uncertain.

As he puts it,

It is our constant efforts to eliminate the negative – insecurity, uncertainty, failure, or sadness – that is what causes us to feel so insecure, anxious, uncertain, or unhappy.

And an especially salient reminder:

The deep truth about insecurity: it is another word for life. That doesn’t mean it’s not wise to protect yourself, as far as you can, from certain specific dangers. But it does mean that feeling secure and really living life are, in some ultimate sense, opposites.

This sentiment was echoed by Rachel Ward in her essay, I’m Sorry I Didn’t Respond to Your Email, My Husband Coughed to Death Two Years Ago:

being comfortable being uncomfortable is a very effective way to be a human.

These have become comforting words, in a way, as I find my place in a new place.

When it comes to the work I do, Seth Godin has some good advice as well. Self-confidence can be hard to come by at times, so he suggests an alternative to prop yourself up when that’s the case: Do work you can believe in. He continues:

Not trust, verification. Not believing that one day you’ll do worthwhile work. Instead, do worthwhile work, look at it, then believe that you can do it again.

There are times when verification isn’t possible, such as when we trust other companies with our information. Data collection matters because of the trust that you and I place in the devices, apps, and services we use and the websites we visit.

Well, Welcome to the Speakularity, Where Everything You Say Is Transcribed and Searchable. Advances in artificial intelligence and natural language processing mean that while systems work hard to anticipate your needs and understand you as a user, they are collecting massive amounts of data about you. Massive amounts of data such as everything that you say to it, or around it. (With Apple’s announcements today expected to involve Siri, I’ll probably want to come back and add five more articles to this).

Many people with Amazon Echo devices discovered during an ad that aired during Stephen Colbert’s Late Night debut that their device is always listening.

As the article about the Speakularity insinuates:

Google, Apple, Amazon, and Microsoft are not interested, today, in recording and transcribing everything we say. They are interested in voice as an interface.

In order to get there, though, they might have to record everything that we say (although they aren’t transcribing it, yet). What if, as the article recalls, Matt Thompson’s 2010 forecasted Speakularity came to pass, and “At some point in the near future, automatic speech transcription will become fast, free, and decent”?

As Thompson points out, it would be a boon to journalism. But it would also be a new, and perhaps disturbing addition, to our digital identities which already suffer from a similar type of auto-archiving. That archiving, self-induced to a point, is our online selves. Social media and other services incentivize data sharing. Facebook, Twitter, LinkedIn, OKCupid—these services are nearly impossible, and rather pointless, to use without sharing personal data. Other sites require data sharing—your bank, your health insurance, your internet service.

What does this mean for you? Well, first of all, you might as well start behaving as though everything you say online is public. As Quinn Norton succinctly puts it, coining “Norton’s Law,”

Over time, all data approaches deleted, or public.

Either your data fades gracefully, gratefully into the ether (see: link rot) or it lives on, forever archived (see: the internet archive). And when it is archived and/or stored somewhere, it can always be found (and thus, always be made public if it isn’t already).

Navneet Alang goes long on the Terror of the Archive for Hazlitt magazine, in the context of the recent Ashley Madison data leak.

the leak has nonetheless forced people to comb back through their pasts and past transgressions. It is an effect of how the web and digitality often collapse the distance between past and present: The archive of who we are in the collection of tweets, status updates, blog posts, and photographs scattered online looms like some peat bog of personality, always about to gurgle up some perfectly preserved act from our personal history.

I wrote about this a bit in terms of the Boston Bombers in A Narrative of Bits and Pieces, but the archive of these moments goes beyond the impression that our online selves give others, and further to disrupt our internal notions of who we are.

The digital archive of the self makes that smoothing effect of time harder to maintain. Those confounded “Memories” or “Flashback” functions in services such as Facebook or Dropbox vomit up images and words from the past at inopportune times—some picture of another ghost to whom you could have been kinder. Meanwhile, Google and Twitter searches dredge up unwanted shards of the self—whether by you or, worse, others with ill intent.

He returns to this concept toward the end of his piece (I’m excerpting a lot here, but I promise, there are so many more good bits like this):

Freud said that we endlessly repeat past hurts, forever re-enacting the same patterns in a futile attempt to patch the un-healable wound. This, more than anything, is the terror of the personal, digital archive: not that it reveals some awful act from the past, some old self that no longer stands for us, but that it reminds us that who we are is in fact a repetition, a cycle, a circular relation of multiple selves to multiple injuries. It’s the self as a bundle of trauma, forever acting out the same tropes in the hopes that we might one day change.

In a nutshell,

To be a digital human is to be many, and in more than one place at once. Am I a good person? Depends on where on the timeless web of self you stand in order to look and judge.

When it comes to memory, our offline self creates our own truth and self-narrative, but the digital archives, well, they don’t forget. Who we are online (forever) isn’t the same person as we think we are offline (now).

That distinction isn’t easily drawn, even in a time of histrionic digital dualism. “Digital dualists believe that the digital world is “virtual” and the physical world “real,” but in this case, the digital world might just be more “real” than the physical world. But it’s important to remember (and easy to forget) that the digital world isn’t the whole story, something that people like Justine Sacco know all too well. Our digital selves aren’t our whole selves, but they are preserved far longer.

So what can be done, with all this data roaming around about us on the web, and on company’s servers? A few things come to mind.

Companies need to get stronger on cybersecurity in a smart way. There is a new cybersecurity mantra, as brought to light by Richard Bejtlich: “If you can’t protect it, don’t collect it.” The less data a company collects about its users, the less data that can be hacked and stolen.
Websites and apps need to seriously curtail their “experience-enhancing” privacy intrusions used to “personalize” your experience by collecting massive amounts of data when you visit a site or use an app. Reducing the amount of analytics and data tracking that these sites and apps perform will not only speed them up (thus actually improving the user experience), it will be a boon to privacy and security as well.
And of course, only share information with Norton’s Law in mind: “Over time, all data approaches deleted, or public.”

Your music recommendation this time around is RAC, with the song Can’t Forget You (ft. Chelsea Lankes). Unintentionally fitting the discussion of digital archives and the permanence of your personal data online.

Thanks are due as well to Audrey Watters’ newsletter, for linking to Quinn Norton’s and Navneet Alang’s pieces, and to Assaf Arkin’s newsletter for sharing the “don’t collect if you can’t protect” piece.

Sewing and the Supply Chain

Thu, 07 May 2015 19:13:36 +0000

I’ve recently started sewing again. After learning how in home economics in junior high school, I decided to pick it up again in order to have an offline hobby with more tangible results (like baking, but longer-lasting). Sewing has changed in the last twenty years, and a fabric wholesaler has witnessed the changes in his own warehouse.

Christopher Higgins owns Globe-tex, a crammed-to-the-gills warehouse in Montreal that’s open to the public. With twenty years of experience, he has seen the fabric industry change dramatically over the course of his career: “Free trade and globalization have totally changed the market. It’s caused a lot of pain for a lot of people and it’s decimated the manufacturing sector altogether. We’ve lost our mills, the technical experts who operate the mills, the chemists who understand dyeing, the engineers who know how to design the textiles.” Since most fabric and clothing manufacturing moved overseas in the past two decades and dropped dramatically in price, home sewing dropped off as well. Sewing used to be the way to build a wardrobe without spending a lot of money, but it has now become more affordable to buy than make. Why sew a dress when you can snag one for $15 at the mall? An overwhelming amount of cheaply made clothing is available everywhere, but ironically, it’s encouraging people to return to sewing once again. We miss quality. We want to feel creative and connected, and to avoid wearing clothing made in dangerous and exploitative sweatshops. We are a movement, and the fabric industry is catching up to us, slowly but surely.

Garment-making has moved primarily overseas to countries like Bangladesh, where working conditions fail workers in the worst ways. Researchers at NYU are attempting to map all of the clothing-manufacturing factories in Bangladesh to improve safety:

A comprehensive, transparent database and map of manufacturers—whatever their size—would begin to make Bangladesh’s vast universe of garment manufacturers visible to auditors, regulators, banks, and brands. To be sure, even many factories that do appear in public databases are hurting for resources to make them safe and stable workplaces.

As globalization has taken over the clothing industry, the price of ready-to-wear clothes has dropped substantially.

Overall, clothes have been getting cheaper for decades, ever since apparel manufacturing started moving to developing countries, where production costs are significantly lower. In the US, the world’s largest apparel market, 97.5 percent of clothing purchased is now imported, according to the American Apparel & Footwear Association. That percentage has risen steadily for years. As recently as 1991, it was just 43.8 percent.

The price of the clothes is often at odds with the condition in which they’re sewn. It wouldn’t take much of a price increase to ensure that the factory workers at the bottom of the chain were well-paid and worked in safer environments, but part of the issue is how far down the supply chain that price increase would have to wend:

The sad part is that the price of individual garments would not have to go up much — 1 percent to 3 percent, various estimates say — to provide a living wage and safer conditions for all those cutting and stitching what we wear. The cycle could slow or even stop. But that 1 percent to 3 percent would have to wend all the way down that river of production — past the eddies and breakwaters of corporate boards and middlemen, subcontracting agents and compradors, to reach those who really need it.

As it gets cheaper to buy clothes, its easier to buy more of them. But what happens after the trends pass? You donate your clothes to a thrift store, or another organization, and then what?

About 10.5 million tons of clothes end up in American landfills each year, and secondhand stores receive so much excess clothing that they only resell about 20 percent of it. The remainder is sent to textile recyclers, where it’s either turned into rags or fibers, or if the quality is high enough, it’s exported and cycled through a cutthroat global used-clothing business.

What doesn’t make it into the stores is then recycled–where else but back in the countries that manufacture the clothes. This 15-minute video follows the work of women responsible for processing the bales of recycled clothes sent over from countries like the United States. Recycled clothes are most often turned into fabric of varying quality–starting the whole process over again. A wholesale fabric retailer estimates that “approximately 80% of fabric in stores is recycled from ready-to-wear”.

Another challenge in the fashion, fabric, and clothing industries is the environmental impact of what you’re buying (true in most industries, honestly). I follow a bunch of sewing bloggers, and they’re often excited about bamboo rayons, and how eco-conscious the fabrics are. However, according to the Federal Trade Commission, that’s all a big marketing ploy.

Extracting bamboo fibers is expensive and time-consuming, and textiles made just from bamboo fiber don’t feel silky smooth. There’s also no evidence that rayon made from bamboo retains the antimicrobial properties of the bamboo plant, as some sellers and manufacturers claim. Even when bamboo is the “plant source” used to create rayon, no traits of the original plant are left in the finished product.

Beyond fiber content, the production of fabrics is challenging to keep environmentally safe. A recent program in China led by the Natural Resources Defense Council’s program Clean by Design aims to help textile mills achieve some of the quick wins of environmentally-friendly textile production. Despite the successes of that program, it will only be harder to improve the environmental processes of the mills beyond these quick wins:

there’s a significant problem that the program can’t easily fix: the use of hazardous chemicals. When the mills dye fabrics, their business depends on getting the colors right for their clothing label clients—which is why they’re reluctant to change their practices. The program also doesn’t address a fundamental problem: the sheer amount of textiles it takes to create the massive volume of clothing the mills produce. Responsibility for that falls on the retailers placing the clothing orders, and it’s one of the biggest issues fashion faces where sustainability is concerned.

And now we’re back where we started. This should make me a more ethical consumer of fast fashion, but so far it hasn’t changed my buying habits too much. I subscribe a bit too much to retail therapy, but I did recently bring several pairs of shoes to be repaired instead of simply throwing them out and replacing them. Slowing down the cycle where I can.

Sewing is just one example of how globalization has affected working conditions, and our knowledge of the supply chains of the products that we buy. It isn’t possible anymore to simply say “Made in America” - often times, products sold as American-made these days, are actually just American-assembled. A lack of clarity about where the things that we buy are made, and who makes them, is troubling.

There are pockets where globalization is clearly not working how you might expect. America imports its own fish, because it is somehow cheaper (due to globalization) to send American-caught fish to China for processing, and then ship it back to America to be sold.

It’s also led to class divisions in terms of what is affordable and desirable.

For most of history, the poor would have eaten the local pigs and known the origin of their socks, and the rich had better access to a global marketplace. But changing elite tastes and the relentless efficiency of supply chains have slowly inverted tastes: In many categories, the poor now buy from the exotic unknown, and the rich insist on what can be traced, from the pig next door to the locally sewn sock.

This is a pattern that you can see in home sewing as well. It isn’t poor women that are returning to sewing, but more well-off women, seeking out high-quality fabric and clothes for themselves that grant the satisfaction of knowing who made an item. Of course, this dichotomy wasn’t supposed to happen with globaliztion (at least according to Jeffrey Rothfeder in the Washington Post).

It was supposed to act like a rising tide, lifting all boats in poor and rich countries alike. Buoyed by hundreds of thousands of new assembly line jobs courtesy of multinationals in emerging nations, the middle class would swell, which in turn would propel higher local consumption. More factories would be needed to meet the demand, further raising local standards of living and handing the largest non-domestic companies a vast and enthusiastic new customer base.

Of course, what happened instead is that the rich got richer. Free trade deals and low labor costs (and safety regulations) abroad meant that countries like the United States benefited more than the countries that took over the labor and factories. Countries like China, Bangladesh, and Pakistan.

Meanwhile, in the United States and Europe, consumers would have their pick of inexpensive items made by people thousands of miles away whose pay was much lower than theirs. And in time trade barriers would drop to support even more multinational expansion and economic gains while geopolitical cooperation would flourish.

Capitalism. Oppression. Globalization. So what can we do? In 2006, Jason Kottke proposed the idea of a True Cost for an item.

Wealth doesn’t just magically materialize into your bank account. It comes from the ground, human effort, the flesh of animals, the sun, and the atom. The global economy is driven by nature, and yet it’s not usually found on the accountant’s balance sheet. Perhaps it should be. I’d like to know the true cost of the stuff I buy. Embodied energy and carbon footprint calculations are a good start, but it would be nice if the product itself came with a True Cost number or rating, like the nutritional information on a cereal box or the Energy Star rating on a refrigerator.

We’re still not there. Some apps on the market make an attempt to add transparency to the global supply chain, but that information is hard to come by for the apps, and hard for the average consumer to find out about. An app can seem like a simple solution, but access to apps isn’t equal (and even among those with smartphones, storage space is limited and attention is lacking).

Now that you feel guilty about all the clothes you own and products that you buy, and the economy as a whole, listen to this song. The band WATERS with their song What’s Real. Probably how you feel about the mall right now.

Advertising Alternatives: It Pays to Be a Google Contributor

Fri, 24 Apr 2015 19:20:57 +0000

Earlier this week I got an email from Google.

One of my principles is to pay for things that I support. I can afford it, and things on the web are relatively cheap. Subscribing to ThinkUp, Pocket Premium, Feedly Pro, each cost about the same as a new pair of shoes, or a nice pair of jeans. To me, that’s a justifiable cost, so I pay it to keep the things I use and love alive.

So when Google Contributor was introduced, I was immediately interested. I hate advertising, but I know it’s one of the few ways to make a living on the web, so I don’t use ad blockers. I have Flash set as click-to-play for security reasons, which turns the worst advertising into gray puzzle piece blocks, but banner ads and others still follow me around the web. After signing up for Google Contributor, more of those ads are replaced with this:

Recently appearing at the top of a Slate article that I’m reading for the next edition of my next newsletter, it’s a welcome relief from ads on the web. The only version of this I’d witnessed before was on one of my favorite music blogs, The Burning Ear. Newly redesigned, the homepage used to look like this (courtesy of the Wayback Machine): Nothing greater than expecting to see a banner ad, and instead seeing “this is an ad-free website. please enjoy this white space.”

Edit: I notice the presence of ads now that they’re no longer (often) advertising. From a Washington Post article:

Accessibility, Sound, and Communication

Tue, 07 Apr 2015 17:58:06 +0000

My birthday was yesterday! To celebrate, I ate an overly large and overly expensive steak and sorely undercooked brussels sprouts. Do yourself a favor and always roast brussels sprouts until they are caramelized and crunchy, then put some reduced apple cider and maple syrup on top. YUM!

Technology, while making the world more accessible than it has been in the past, has a lot of work to do for people with disabilities. A huge example of this is the shortcomings in OCR (optical character recognition) technology. In short, OCR sucks. And when we use it to simplify our lives (make a PDF into something that I can copy-paste into a text file), then when it fails it’s a minor inconvenience, and a silly one at that. Just one problem. If you depend on OCR to read your college textbooks, and the accuracy of OCR has a direct bearing on your understanding of a subject, then it isn’t so funny anymore. And acknowledging how terrible it is doesn’t mean anything will change. As the essay states,

“The only thing that I’ve ever seen quickly propel a piece of adaptive tech from mediocre to genuinely useful is adoption by nondisabled people.”

For a glimpse into just how badly OCR works at times, the Reverse OCR tumblr showcases random scribbles that OCR has identified as words.

Mel Chua has an excellent blog with an ongoing series documenting what “being deaf is” for her. For example, Being deaf is: constant baseline soundmaking anxiety. Since she’s deaf, she doesn’t know when her activities are making noise, which can be an issue. As she puts it,

“When the world around you doesn’t give you feedback, and the best gauge you have of your own noise level is frustration on the faces of the people near you, then sometimes you tiptoe through the world.”

Designing things for the blind is easily overlooked, but with some work, is easy to remedy. Like most design, it takes some extra thought, user testing, and novel approaches. Maps of public places, for example, are something that aren’t often accessible to the blind. Google maps and spoken directions can help some, but not indoors. That’s where people like Dr. Joshua Miele come in–to design maps for the blind.

Designing buildings for the blind also takes a unique perspective, one that architect Chris Downey gained when he lost his sight. Flooring is important, and in a project “he used different kinds of flooring—cork and polished concrete—to help the visually impaired differentiate offices from common spaces.”

Vancouver enshrined such accessibility-based design in its building code by banning doorknobs. A seemingly random, arbitrary step, Vancouver recognized that doorknobs didn’t meet accessibility guidelines. Using a door handle or lever allows better grip for people with limited use of their hands–because their hands are full of groceries, they have arthritis, or they simply don’t have hands.

For most people, sound is integral to their experiences of the world. Therefore, maybe it shouldn’t have surprised me to learn that sound design doesn’t just happen in movies. It also happens when your car is built.

“For most people, sound is more than a set of vibrations; it carries complex social and cultural signals. This point has not been lost on designers, and thanks to new digital tools and the insights of big data, our soundscapes are being crafted with extreme precision. Engineers can manipulate the sounds our products and gadgets make so that they do more than please our ears — they influence our perception.”

It’s not just the sound of your car door closing, but the sound of your engine too that’s being designed.

Efforts to make the invisible more visible (or perhaps, the quiet things louder) can mean experiments in giving wi-fi a sound:

“As the user walks through the city, a hacked iPhone sniffs out nearby Wi-Fi. The interface identifies traceable characteristics like locations and data rates, and translates everything into sound using a pre-programmed language of parameters. The sound is then transmitted to a modified hearing aid, creating sonic representations all around the user.”

Wi-fi sound art experiments aside, the Internet really does make noise. In server farms, the noise of thousands of whirring computer fans personifies the Internet’s sonic profile. If you click through, you can listen to it.

Communicating to the public is a high priority when natural disasters are approaching. Winter storms, tropical storms, hurricanes, earthquakes, and so many others require communication in a stressful time. That’s why the National Weather Service (NWS) takes special care to draft their pre-storm warnings.

“Getting the messaging right is so important that the NWS is consulting with social scientists and risk communication experts to revise how it issues warnings so that people pay attention to them. When the coverage and the messaging is so uniformly alarming, as it is today, it’s not that much of an issue. But those warnings issued for a few inches of snow is where the agency is trying to improve.”

The Weather Channel, meanwhile, has begun naming winter storms (since last season or so, I believe). This is a bad idea, and here’s why:

“TWC, meanwhile, names winter storms suddenly, whenever they feel like it basically. These winter storms, particularly Nor’easters, often come and go in a matter of a couple days, not really enough time to heighten awareness and inspire storm preparation.”

Compared with the National Hurricane Center, and others who spend hours determining how to classify storms, The Weather Channel does it for ratings. I am so glad winter is over. We’re anxiously tracking the progress of flower buds in the woods behind my office, and I am delighted that my new raincoat is getting some use this week–if only because the rain means it’s too warm to snow!

Finally, the best thing I’ve found on the Internet lately. Is the Internet getting you down? Do you wish every meal could be brunch? Use this JavaScript bookmarklet to turn every image on a webpage into an image of french toast. #inspired This week’s music is a song you may have heard. Bleachers, with their song, I Wanna Get Better.

Old news to you? Alvvays channels Best Coast in their fuzzed out hit in the indie circles, Marry Me Archie.

Dams, Fish, and Engineering Disasters

Thu, 26 Mar 2015 18:49:09 +0000

David W. Butterfield (American, 1844 - 1933) Vermont and Massachusetts Railroad, about 1870 - 1880, Albumen silver print 42.2 x 56.6 cm (16 5/8 x 22 5/16 in.) The J. Paul Getty Museum, Los Angeles

Once upon a time (okay, the first one was in 1871), fish were carted across America in railroad cars. Of course, this was terrible for native fish species, but great for fisherman looking to fish fish that were in demand.

More than a century later, a community around the Putah Creek in California rallied together to restore a creek that had run dry after the watershed was transformed by a dam and opportunistic farmers.

Dams have a way of transforming rivers, but as the film Dawn of the Planet of the Apes makes clear, hydroelectric dams generate both vital and renewable energy for modern civilization.

People’s needs usually trump those of other animals, so fish get the short end of the…river. Routing fish around dams to allow them to continue their breeding patterns is a challenge, leading to the implementation of fish ladders (or cannons) to ensure that the fish can get where they need to be.

Carleton Watkins (American, 1829 - 1916) [Benton Dam, Merced River], 1860, Salted paper print 34.8 x 41.9 cm (13 11/16 x 16 1/2 in.) The J. Paul Getty Museum, Los Angeles

Maintaining and running a dam is a massive undertaking, especially when the dam(s) in question control the flow of the Columbia river.

“But as large as the dams are, their margins are minuscule and operating them takes unerring foresight and subtle management: let too much water fill reservoirs and a rainstorm might flood Portland; keep the reservoirs too empty and you’ll parch farmers. Send too much water over a dam’s spillway and you’ll suffocate fish with dissolved gases; send too much through its turbines and you’ll overload the electrical grid.”

Hydroelectric power is fascinating and powerful, and involves a lot of math and careful calculations. How many fish make it through the dam is part of this:

“fish throughput is carefully counted by attendants sitting in darkened rooms behind windows at the dam’s fish ladders—3,939,524 fish swam upstream past Bonneville Dam in 2010, including 809,512 adult Chinook salmon and 11,183 lampreys. (On their way upriver, fish go through the ladders; heading downriver, they go over the dam’s spillway, through its turbines—where, astonishingly, 96% of them survive the trip—or through a bypass tube that spits them out two miles below the dam.”

Dams are especially hard to manage, with so many competing resource demands:

“it must generate hydroelectricity, conserve fish, provide shipping channels and recreational areas and irrigate farmland. And all of those goals are directly opposed to flood control in the way that reservoirs are operated, calling for generally full reservoirs while flood control requires low reservoirs that are ever ready to receive runoff.”

So we have fish in rail cars, fish that need help getting over dams, dams that generate electricity, and the fact that hydroelectricity is a really excellent renewable source of energy.

Hydroelectricity is known as “flexible dispatch” electricity because the amount of energy produced can shift with a moments notice. Hydroelectric power

“is the biggest source of flexible dispatch available—and is much cheaper to run than the natural-gas turbines that serve that market in other parts of the country.”

Carleton Watkins (American, 1829 - 1916) [Rudyard (English) Reservoir. Central Dam 114 feet high; Timber Crib and Stone Filling], about 1871, Albumen silver print 41.3 x 54.9 cm (16 1/4 x 21 5/8 in.) The J. Paul Getty Museum, Los Angeles

Flexible dispatch is an important factor of energy creation, because the electric grid needs as much help as it can to cope with peak demand times–the morning, when people get ready for work, and the evening when people return from work.

Solar power isn’t as good of an alternative for managing peak demand, since it relies on the sun and the direction that the panels face. That direction also plays an important part in managing the electric grid. While facing your solar panels south lets them generate maximum energy, facing them westward may help them generate energy at the times the electric grid most needs it.

It doesn’t take much to cause an electricity outage in the United States. A blackout in the Northeast United States in 2003 happened because three things went wrong. Inconsequential on their own, together they were catastrophic–they caused the largest blackout in America.

“Around 12:15, on the afternoon of August 14, 2003, a software program that helps monitor how well the electric grid is working in the American Midwest shut itself down after after it started getting incorrect input data. The problem was quickly fixed. But nobody turned the program back on again.

A little over an hour later, one of the six coal-fired generators at the Eastlake Power Plant in Ohio shut down. An hour after that, the alarm and monitoring system in the control room of one of the nation’s largest electric conglomerates failed. It, too, was left turned off.”

Disasters can happen surprisingly easily. As systems get more complex, it often gets more difficult to diagnose what could be causing an issue.

Fédèle Azari (Italian, 1895 - 1930) [Crashed Caproni Ca. 36], 1914 - 1929, Gelatin silver print 11.8 x 16.6 cm (4 5/8 x 6 9/16 in.) The J. Paul Getty Museum, Los Angeles

Although the cause of the recent tragedy of the crashed Germanwings flight 4U9525 is still under some dispute, plane crashes are often caused by failures of systems and/or processes. In 1983, a plane in Canada thankfully did not crash after both engines failed and ran out of fuel mid-flight. Four breakdowns in communication and decision-making led to that near-disaster.

A 1931 air disaster is helpful to examine in terms of digital security:

“First, few seem to care about disasters involving new technologies until a celebrity dies. While no one would like to see such an event occur, it’s possible real change of opinion and technology will not happen until a modern Knute Rockne suffers at the hands of a security incident. Second, authorities often do not have a real incentive to fix processes and methods until a tragedy like this occurs. Out of this incident came pressure to deploy flight data recorders and more robust aviation organizations. Third, real inspection regulations and technological innovation followed the crash, so such momentum may appear after digital wrecks.”

That could explain the influx of privacy- and cybersecurity-oriented bills that we’re seeing in Congress this year, in light of the high profile Target data breach and Sony Pictures Entertainment hack.

The most notable difference between disasters in the “real” world compared to the digital world is that:

“everyone knows about it. The disaster is visible, and engineers can analyze and learn from the event. The lessons they take away make future bridges stronger and safer. I do not see this happening in the digital world. Organizations suffer disasters all the time due to poor techniques, tools, configuration, management decisions, and so on. Unfortunately, few people ever hear about these problems, so they are repeated elsewhere. The only parties to benefit are intruders. Security engineers never get to learn from the mistakes of others.”

Here’s hoping some infrastructure funding and disaster recovery planning continues to be a part of America’s future. (Along with fish cannons.) All images courtesy of the Getty’s Open Content Program

Celeste LeCompte shared the first two links in Maggie Koerth-Baker’s newsletter, The Fellowship of Three Things

Prescriptive Design and the Decline of Manuals

Tue, 24 Feb 2015 18:35:24 +0000

Instruction manuals, and instructions in general, are incredibly important. I could be biased, since part of my job involves writing instructions for systems, but really, they’re important! As this look into the historical importance of manuals makes clear, manuals (and instructions) make accessible professions, tools, and devices to anyone that can read them (which, admittedly, could be a hurdle of its own):

“With no established guild system in place for many of these new professions (printer, navigator, and so on), readers could, with the help of a manual, circumvent years of apprenticeship and change the course of their lives, at least in theory.”

However, as the economy and labor system shifted, manuals did too:

“in the 1980s, the manual began to change. Instead of growing, it began to shrink and even disappear. Instead of mastery, it promised competence.”

And nowadays, manuals are very rarely separate from the devices or systems they seek to explain:

“the help we once sought from a manual is now mostly embedded into the apps we use every day. It could also be crowdsourced, with users contributing Q&As or uploading how-to videos to YouTube, or it could programmed into a weak artificial intelligence such as Siri or Cortana.”

By attempting to make things more seamless, integrated, and easy-to-use, devices not only technologically lock out users with copyright and DRM and pre-soldered hardware, they also provide short, “quick start guides” in place of a true manual before you can begin using a machine.

“Furnished only with a manual of one or two pages, users soon reach a comfort zone, a knowledge plateau from which they tend not to wander. The aggregate effect, culturally, may be that less is less. The less we’re inclined to know about our devices, the more beholden we are to the manufacturers that make them, and the more we offer control to those who, for good or for ill, know more than we do. If manuals began as great equalizers, then their disappearance should at least give us pause. By dispensing with them, we could, consciously or no, be setting the stage for something few would relish: a society divided.”

This is certainly preferable by a large portion of people (judging by the types of calls and help requests I would get during my years of customer support). People want devices to just work, so they can move on to what they need to do–buy groceries, file their taxes, and yes, write documentation.

The trouble is that we’ve built a near-incomprehensible technological system, and now plan on using that technology in nearly every device on the market today. The internet of things is here, it doesn’t always work right (video), and the design isn’t usually easy to understand.

Design, when algorithmically-controlled to the point of being a total black box to a person, isn’t exactly helpful. In those cases we’re stuck wondering, Why is my house this temperature? Why is Facebook showing me this? Why am I being shown an ad about asphalt mixtures?

The rise of the algorithmically-sorted life isn’t exactly welcome, even if it purports to be helpful and make things easier to use (okay maybe the asphalt mixture ad is under no such illusions). However, that doesn’t necessarily make all prescriptive design bad (or do you think it does?). Some prescriptive design is basic and helpful–make it easy and clear what the person should do next.

Google has started using insistent, prescriptive design when it comes to various kinds of security warnings.

Gmail warns for suspicious email.
Search blocks “crapware” and malicious software download sites from appearing in search results.
Chrome blocks malicious software from being successfully downloaded at all.
Chrome improved its warnings for SSL (secure sockets layer). (Oddly, Wikipedia no longer has a dedicated page about SSL, instead directing people to the page for TLS (the encryption method that superseded it). The page now has a warning because it’s too long.)

Google is taking two different steps with these design choices.

In the case of the spam emails, when they are identified as malicious, Google tells you exactly why you should be suspicious of the message–perhaps recognizing that more information in this case could mean more obedience to best practices, e.g., don’t click on a phishing email.
In the case of the malicious software downloads and the SSL warnings, Google takes the step of making it hard for you to accomplish something (click through a warning, download a file, click on a search result) with the end goal of preventing you from doing something harmful (putting your data at risk in a man-in-the-middle attack, having your computer infected with malware or a virus).

This isn’t so bad, as long as the design choices are made with strict user benefit in mind (and to that end, who gets to decide what is beneficial for the user?) As the presentation on the SSL warnings makes clear, the goal is to change behavior, since changing comprehension alone didn’t warrant enough of a change.

Google may be parenting the Internet with their design choices, but in this case, the Internet could use a little bit of growing up (and so could its users).

Writing error messages is an art in and of itself. A guide on how to write error messages identifies that:

“A good error message has three parts:

The problem - explains that an error has happened;

The cause - explains what caused the problem;

The solution - explains how to overcome the problem.”

This is even tougher to do when you need to keep them as short as possible. Error messages today can be overly generic, and even purposely unhelpful in the hopes of making a system more secure. Two types of errors are especially guilty of this:

With a little bit of work, these workflows can be improved. System security can be strengthened without providing vague, unhelpful error messages to a person. I’ve attempted to log in 10 times in a row, getting the “username or password is incorrect” message and re-entering my password (since I can’t see my password when I enter it, it just shows up as little black dots). After 10 tries I finally realized that I had mistyped my email address on the first try, but never thought to correct it (it’s the shorter of the two, after all).

Data Privacy Day!

Wed, 28 Jan 2015 18:15:42 +0000

It may seem like a Hallmark holiday, but there aren’t any cards for it (if there are please mail me one). It exists to commemorate the 1981 signing of the Convention for the Protection of Individuals with regard to Automatic Processing of Personal Data by the Council of Europe. Accordingly, Europe celebrates this day as Data Protection Day.

Let’s start with some basics…

Use secure passwords, and store them securely. Memorize them, keep a list in your wallet, or use a password manager like LastPass, KeePass, or 1Password. Don’t store them in your browser.
Don’t get caught by phishing scams. Think before you click on hyperlinks and email attachments, especially on a mobile device. Quiz yourself to see if you’re prepared.
Secure your mobile devices. Limit the information that you send to Apple, Google, and the apps you install. At the least, set a passcode, opt out of ad tracking, and manage which apps have access to your location.
- Tips for Android Phones
- Tips for iPhones/iPads

A few more steps that are good to take…

Delete your profiles from data broker sites. Your data is valuable to advertisers and criminals.
- See how much it’s worth (mine is worth $0.76).
- Watch this short (1:40 mins) video to see how much you may share.
Delete accounts you no longer use. Seriously, if you haven’t logged into it for over a year, consider whether you still need it. Dormant accounts are a security risk.
Avoid using public wi-fi and USB charging stations. You don’t know who set up the network or the cables, and your personal data could be stolen from your device.

And to keep protecting your privacy…

Don’t use services that abuse your privacy. Spend money on services that protect your privacy, unlike services like Uber, that track you incessantly and seek to upend the relatively anonymous form of travel by taxicab.
Manage browser cookies, and beware supercookies. Use a Do Not Track setting when available, and clear your browser cookies periodically. Eat baked cookies.
Consider your posthumous digital privacy. Establish a plan for what will happen with your digital accounts after you die. It should be done in advance, to avoid difficulties for your heirs.

Don’t try to do all of this at once, but try to make sure that you do it occasionally. Add a bunch of things to your to-do list, make recurring calendar events to update passwords, revoke third-party app access, and check your credit report.

The Federal Trade Commission has a ton of information about how to protect yourself. Read up, stay informed, and encourage your friends and family to do the same!

We’ll return to regularly scheduled silliness later this week. Go eat (and delete!) some cookies.

Libraries, Digital Advertising, and the Machine Zone

Fri, 16 Jan 2015 20:28:23 +0000

Librarians are an underused, underpaid, and underestimated legion. And one librarian in particular is frustrated by e-book lending. Not just the fact that libraries have to maintain waitlists for access to a digital file, but also that the barriers to checking out an ebook are unnecessarily high. As she puts it,

“Teaching people about having technology serve them includes helping them learn to assess and evaluate risk for themselves.”

In her view,

“Information workers need to be willing to step up and be more honest about how technology really works and not silently carry water for bad systems. People trust us to tell them the truth.”

That seems like the least that can be expected by library patrons.

We are all users of technology, and we can demand more of it and its creators rather than accepting the software and application environment that we currently inhabit. (The indie web movement I mentioned last time would certainly agree).

E-books are part of the growing shift in the meaning of ownership. As Alexis Madrigal and Rob Meyer explore at The Atlantic, when everything works like your cell phone and is connected to the internet, “three things happen: it becomes smart, it becomes hackable, and it’s no longer something you own.”

Joining libraries as common, valuable, and strapped-for-cash cultural institutions are museums. Museums are starting to make more use of data that they gather by tracking customer behavior. Sensors to monitor patron movement, demographic analyses–not always new techniques, just automated ones. This has serious privacy implications, but is all-too-common (retail stores are doing the same and beyond). As museums continue to focus on data to gather insights on the populations that they serve, they need to make sure that they’re using data in the right ways, and are prepared to adapt.

The internet advertising industry had a bit of a wake-up call last month. Google shared the results of a study they conducted on the viewability of Google Ads. Stunningly, “many of the ads served on the web never appear on a screen.” That doesn’t mean that they weren’t loaded on the page, but that they didn’t meet the criteria for viewabiity as defined by the Media Rating Council: “A display ad is considered viewable when 50% of an ad’s pixels are in view on the screen for a minimum of one second.” Even worse, another study by security researchers and the Association of National Advertisers revealed that:

“Almost one-fourth of video ads and 11 percent of display ads are viewed by fake consumers created by cyber crime networks seeking to take a chunk of the billions of dollars spent on digital advertising, according to a new research report released on Tuesday.”

Say 100 digital ads are purchased. On average, half of them (50) will be viewed. Of those 50, 11% (5.5) are viewed by bots, not real people. #botsruntheweb

In writing my lead-in to this next link, I ended up writing about my resolutions/goals for this next year. Because have you ever found yourself on Facebook (Twitter, Tumblr), and then look at the clock and realize that an hour has passed? Or that you’ve clicked through to your cousin’s girlfriend’s sorority sister’s boyfriend’s profile? You don’t even know or care about these people, and yet you’re staring at their profiles, clicking and scrolling. You’ve hit the machine zone.

“What is the machine zone? It’s a rhythm. It’s a response to a fine-tuned feedback loop. It’s a powerful space-time distortion. You hit a button. Something happens. You hit it again. Something similar, but not exactly the same happens. Maybe you win, maybe you don’t. Repeat. Repeat. Repeat. Repeat. Repeat. It’s the pleasure of the repeat, the security of the loop.”

That loop is exactly why I made resolution #1, which is to stay off Twitter more and read fewer articles on the web. The machine zone is just that—a zone within a machine, not within a genuinely social context.

“It explains how the more Facebook has tuned its services, the more people seem to dislike the experiences they have, even as they don’t abandon them. It helps explain why people keep going back to services that suck them in, even when they say they don’t want to.”

It isn’t fun like it used to be (if it ever was):

“on occasion, on those very same days I’m supposedly too busy to go out, I spend several hours clicking around to random stuff online. “It would be one thing,” I was telling my husband a couple months ago, “if I got to the end of the day and thought, well, I squandered every free second I had on the Internet today, but at least I had a really great time!” I remember when I did feel this way, more than a decade ago, when I was first blogging, but I never do now. I think about how much nicer it would have been to go out and talk to someone.”

The Oscar nominations were announced recently, and critics were quick to point out that many well-regarded movies, like Selma and the Lego Movie, were snubbed. Not only that, but this is one of the least-diverse Oscar nominations field in years. The Atlantic ran the numbers of how many films with Best Actress nominees are also nominated for Best Picture, and it’s far lower than the number of films with Best Actor nominees. This year’s Oscar field is no exception.

A new antibiotic was discovered, thank GOODNESS.

“A team of scientists led by Kim Lewis from Northeastern University have identified a new antibiotic called teixobactin, which kills some kinds of bacteria by preventing them from building their outer coats. They used it to successfully treat antibiotic-resistant infections in mice. And more importantly, when they tried to deliberately evolve strains of bacteria that resist the drug, they failed. Teixobactin appears resistant to resistance.”

That’s a promising sign, but we’re still making mistakes in how we use antibiotics in the U.S.. Not only are they overprescribed for sicknesses like the common cold (a virus), but they’re also (over)used to treat adult acne.

“Antibiotics are one of the main ways to treat moderate to severe acne, and patients often are put on them for months or years. Although dermatologists represent only 1% of the nation’s physicians, they prescribe 5% of antibiotics, pharmaceutical-industry data show. Over time the microorganisms the antibiotics are designed to kill adapt to them, making the drugs less effective."

While acne is pretty awful, there are alternative methods that are more effective, but are historically frightening, like the drug commonly known as Accutane. Accutane is packaged in a drug packet with frightening diagrams of the babies that can result if you get pregnant while on the drug, plus you have to take a monthly quiz to prove that you understand the risks if you get pregnant, including declaring two different forms of birth control that you use. I’d prefer that to contributing to wide-scale antibiotic resistance any day!

I’m overly ambitious with themed links, contributing to the slow, but still stay tuned to hear more about river reconstruction, container ships, high frequency trading microwave networks, and much more!

I’m listening to Daughter while I write this, and making my way through a book about HTML and CSS to make sure I fill any holes in my self-taught knowledge before I move on to JavaScript (there’s that new year goal kicking in). We’ll see how it goes.

So This Is The New Year

Thu, 08 Jan 2015 19:45:50 +0000

I didn’t start this as a resolution post, but here we are. It’s easier to write the introduction after the essay is written, so here I am to tell you this is a post of my 2015 resolutions. This year is all about purging the “someday maybes” and turning ideas into actions. Taking care of myself and moving forward.

1. Stay off Twitter more, read fewer articles on the web, and create more.

I came back from the holiday break with a renewed apathy about social media. I’m a webaholic in the worst way, but vacations and trips where I don’t live on social media seem to be more refreshing. When I went to Germany last summer, and on this recent holiday break, I either didn’t bring or didn’t open my computer, and deleted the Twitter app and a few others. I read four books each two-week trip.

Twitter perhaps is the worst app for me, since its endless feed helps you get sucked in. But talking to people that don’t use Twitter, you start to realize just how few people use Twitter, how unrepresentative it is of the actual population. My feed is dominated by tech and media types, and it ends up being quite a circular clique. When news came that the owner of The New Republic had replaced the editor, and most of the masthead resigned in protest, my Twitter feed exploded with the news, posting reactions and opinions all day long. No one I talked to offline had even heard of the magazine. I don’t even read it, let alone care about it. It was time for a change.

So far this year I’ve been on Twitter once a day at most. I’ve retweeted 2 things since the beginning of the year and posted nothing. I don’t miss it. I’m sure I’ll use it more for networking and self-promotion when the need arises, and I enjoy having an outlet where I can interact with people I’d never meet in real life—authors of books I’ve enjoyed, speakers whose talks I’ve watched on recordings—and yet, I need to make room for the people that I’ve not only met, but am friends with in real life. I guess that makes me one of those “unpluggers” but it’s more me acting on a realization. If a friend doesn’t add anything to your life, and only takes away your energy and manipulates your emotion, you should separate from that friend.

So, Twitter, I’m breaking up with you. I’ll use the cowards method and just stop calling you as often, send you a text when I’m intoxicated (with ideas and thoughts), but stop abiding by the demands on my time and attention that you make of me. I won’t be your enabler, I’m cutting you off.

Twitter isn’t alone. I’m unsubscribing from email newsletters and RSS feeds en masse. If I don’t open it, if I don’t consistently get value from the posts (and instead just click through them so the numbers disappear), I do not need them. I usually pile them back up, but I’m hoping to focus more on tangible goals this year. As I was unsubscribing to feeds, I was adding new ones, but ones about sewing or cooking. I feel proud when I write something, like this blog post.

I feel proud when I make something new, like a brand new recipe, or sew something, or I finish a book. I don’t feel proud when I consume content on the web. It feels like an obligation, and fills up an endless keyword repository in my brain (opening articles that I might not find interesting, but someone else might), google docs full of ideas (but mostly links to articles), and a OneTab extension with over 2000 stored tabs. I’m learning, but don’t leave myself time to think.

I’ve always associated part of my identity with being informed. I was notorious in high school for using the phrase “I read somewhere” and asking a question. I was showing off. I also couldn’t remember where I’d read the fact, but always wanted to know more. That won’t change. What will change is the focus. I’m taking back my attention span, and lasering in on what (and who) matters to me.

2. Learn JavaScript

I made a lot of headway on this goal last year. I know HTML and CSS fairly well, and want to flesh out my front-end web language knowledge. It’s slow going, considering that it isn’t a display-oriented language but rather an object-oriented language, but I’m pushing on. I’m focusing on this so heavily because not only is it a practical language to know, it will help me achieve some other, nerdy goals that I have.

I want to learn how to call an API, which is apparently fairly easy to do in-browser, but I have no idea how it works. I have all sorts of applications that I use, have used for years, that have accumulated vast amounts of data on me that I have no way of accessing. Last.FM comes to mind especially. Nearly everything I’ve listened to for the last 7 years is documented in there, but the only way to get that information out to play with is through an API. Let’s do it. What will I do once I get that information? Well, then it’s data visualization time. I’ve been trending somewhat quantified self this past year as well, influenced by the greater social trend of self-tracking but also by a desire to better understand myself.
I want to learn how to make cool data visualizations. Maybe I was super into punk music until college, and then my teenage angst calmed down. Maybe it took a few more years. I’m curious to know how I’ve grown, and it’s easiest to see when directly manipulating and visualizing data that can tell me in ways that my own, unreliable memory can’t.
I want to write something that uses the Tangle JavaScript library. I was transfixed by this when I first read Explorable Explanations (one of the open tabs I read that managed to stay open until the whole article was read). This, this is what the web is all about. Interactive, educational, essays. No more news articles copy-pasted from their paper mockups. Not only can references be made interactive on the web, now the information behind them can be directly manipulated for greater understanding. Especially considering how content on the web is read differently than content on paper, it’s important to have ways to convey long, detailed information in understandable, interactive ways. I want to be a part of that. Essays like Parable of the Polygons show me that it’s truly possible. You can read countless articles about how segregation happens, but somehow seeing it so simply depicted—and interactively understood, the concept becomes so much clearer.

I completed the Javascripting tutorial (which I loved, because it helped me get familiar with Terminal and writing JavaScript in a text editor), and am 71% through the codecademy tutorial. I just need to practice, practice, practice. And keep learning.

3. Read something huge, and write something huge.

I’ve owned Gödel, Escher, Bach: An Eternal Golden Braid for a few years now. I managed to get a few pages into it, but not far. It’s a huge book. I’ve read books that long before, but not quite as dense, especially without guidance (book club, anyone?). I’d like to read it this year. It’s been recommended by many people, but when two of the people I thought least likely to agree on something each independently told me I should read it, I knew it was worthy of reading. Now I just need to read it.

I’ve also had a giant idea lurking, for nearly all of last year. I’ve been using a Scrivener trial to try to make better sense of all the tangents I keep going down. I set up four Google alerts to see if new things were published about the topic. It’s morphed into an idea that I’m collecting links about, instead of writing about. Part of it is that I don’t feel even remotely qualified to be writing it and drawing the types of conclusions that I want to draw. Cue an endless link-hunt. But when I think about it, I still get interested, and excited. It may be a dead idea, with too much going on, but I’m going to try to pull it out of the weeds this year. Optimally it’d involve some independent research involving large-scale web URL analysis. Realistically, I have no idea how to do that and not much of an idea how to approach that kind of project.

So there we have it. My three major goals for next year. Let’s get to work.

Torture, Ownership, and Privacy

Sun, 14 Dec 2014 18:58:31 +0000

The Senate Intelligence Committee released hundreds of pages (soon available as a book) detailing acts of torture committed by the CIA.

The New York Times calls out 7 key points from the report.
The Atlantic takes to task leadership proclaiming that “this is not who we are”, by reminding us that torture is who we are.
Surgeon and writer Atul Gawande tweeted about the role of medical professionals in the torture, and it is truly appalling.
Senator John McCain, a survivor of torture as a POW in the Vietnam War, released a statement condemning the torture committed by the CIA:

“What might come as a surprise, not just to our enemies, but to many Americans, is how little these practices did to aid our efforts to bring 9/11 culprits to justice and to find and prevent terrorist attacks today and tomorrow. That could be a real surprise, since it contradicts the many assurances provided by intelligence officials on the record and in private that enhanced interrogation techniques were indispensable in the war against terrorism. And I suspect the objection of those same officials to the release of this report is really focused on that disclosure – torture’s ineffectiveness – because we gave up much in the expectation that torture would make us safer. Too much.”

Reading about the torture committed by U.S. citizens is appalling and disturbing. I can only hope that real change comes from this, but am not optimistic considering that it happened despite infighting within the CIA, despite previously stating in 1989 to the Senate that torture is counterproductive, yet it still was committed and on a large scale. How many war crimes does the U.S. have to commit before it is charged?

For something distracting and enjoyable to look at, Twitter image bots interact and great creations result. My favorite, CommonsBot, takes creative-commons-sourced images and tweets them at other bots, like pixelsorter or imgshredder, which then transform the images. As my Twitter feed has a tendency to get depressing, or too journalist/news-based at times, these images are a great break in the madness.

In case you didn’t know, essentially everything in the world is politicized. Clocks, those objects on the wall, on your phone, on your wrist, on your microwave. They’re political, especially if you live in Bolivia:

“There are no “universal” laws of time. It’s all up for debate, and it has been for a long time.”

So clocks are political; the result of years of colonial input. What about statistics? Surely numbers remain impartial? Alas.

“the qualitative descriptions used by early modern states to keep track of their subjects and wealth eventually merged with the mathematical theory of probability to bring it about.”

Terms like GDP and other statistics are politicized, and many aren’t even standardizable across EU member states. Stock images too, are political (perhaps less surprisingly).

As big data hypes itself into the national consciousness, people are paying more attention to privacy and data ownership (or at least the media is)… Uber is a fine example, with scandals over the last month about how well they control access to data. Sociologists Zeynep Tufekci and Brayden King discuss the need for additional regulations of data in the light of the fact that we can’t trust Uber:

“We already regulate sensitive data, ranging from health records to financial information. We must update oversight for 21st-century data as well. When we’re picked up on a rainy street corner, it’s not enough to know where the car is going. We need to know where our data is going, and how it’s used.”

When it comes to big data, it isn’t clear who owns the data elements after they become a part of big data. One example is in farming, where Agriculture-Technology Providers are ushering farmers into the era of big data, in the hopes of reducing “uncertainties inherent to farming. However, farmers are hesitant to lose control of their data and want assurances before they sign contracts.” As farming is guided more by big data, the role of automation and quantitative measures on how crops are grown, as well as how that impacts the legacy of family farms, remains to be seen.

“Like many farmers Mr. Tom is wary of what big company might own his data. He shares some information with Monsanto, for example, but is careful of others’ policies around data retention. He also worries about how computation is going to change the farm he hopes to leave to his children.”

Data ownership seems to be one of the prime issues of our times, and its evolution will be interesting to observe, especially in as things like learning analytics are greatly emphasized. Do the students have any right to their data as it is used to assess learning outcomes? Is it governed by FERPA? Who owns the content in Learning Management Systems? More heavy questions I won’t try to answer.

Aside from ownership of data, ownership of your web presence is just as important. It isn’t easy for everyone to maintain, but owning your web presence can be as easy as periodically googling yourself to see what comes up, or using only name-based usernames on services that you want yourself to be discoverable on. For those of us with websites, projects like the Domain of One’s Own project at University of Mary Washington are fascinating.

“A Domain of One’s Own is a project at the University of Mary Washington that, starting Fall 2013, will provide all incoming freshmen with their own domain names and Web space. Students will have the freedom to create subdomains, install any LAMP-compatible software, setup databases and email addresses, and carve out their own space on the web that they own and control.”

I would’ve loved to have had something like that in college, but at the same rate, I’m sure it leads to a lot of abandoned webspaces for some alumni. The indieweb project is also a strong proponent of owning your online identity, as demonstrated in this video geared toward developers.

Ownership of your data, ownership of your web identity, these require what is being called first-person technologies. Technologies created, and owned, by the people that use them (rather than Google, Facebook, or others).

“We won’t get back our privacy, or make real progress toward real personal freedom, until we develop and deploy first person technologies for everybody. Without them our democracies and marketplaces will also continue to be compromised, because both require those three virtues of privacy.”

The three virtues of privacy are secrecy, anonymity, and autonomy.

Privacy is so, so important in today’s world. There are companies whose sole purpose is to collect information about you, create profiles of you, and sell it to advertisers or whoever is willing to pay for it. They’re called data brokers. There is a way out, though, by opting out of these services. One of the hardest parts of maintaining privacy and security on the web is that the tools available to protect yourself are very difficult to use. Privacy mechanisms that trick you into making your posts public by default, or making it difficult or impossible to change the settings of something which has been posted, pages that urge you to complete your profile by adding your location to your profile. Logical and usable computer security is something that needs more attention, and thankfully, seems to be getting some.

Some use cases for usable computer security are compiled by everyone’s favorite infosec heroine, Taylor Swift, and Eleanor Saitta takes it further by examining specific use cases by high-risk users, such as a woman at a domestic violence shelter.

Aside from data brokers, one of the few ways that websites make money is by selling ads. (Other ways include direct-funding models like Patreon, or subscription/paywall models used by the Wall Street Journal, the New York Times, and many other sites.) Thinker Seth Godin offers a simple way to think about the effectiveness of digital advertising–would you miss them if they weren’t there? Most people I know use AdBlock Plus, thus answering Godin’s question–no, in fact, a web without ads is preferable to a web with ads as they currently pervade our online experiences. “By downloading the plug-in AdBlock Plus (ABP) on a massive scale, users do vote with their mice against the growing invasiveness of digital advertising” Its developer is called Eyeo:

“Eyeo GmbH is filling a vacuum created by the incompetence and sloppiness of the advertising community’s, namely creative agencies, media buyers and organizations that are supposed to coordinate the whole ecosystem (such as the Internet Advertising Bureau.)”

An alternative form of advertising, and perhaps one that you might miss if it weren’t there, is something that tumblr is experimenting with: A buy button that appears on posts from certain websites (Etsy and Kickstarter among them). I haven’t seen it show up yet, but it seems like a really great idea to have a non-intrusive, practical form of “advertising” on the web. Another idea is Google’s Contributor, which as of now is an invitation-only experiment that involves paying a few dollars a month to remove ads from participating websites (like Mashable, the Onion, and imgur). I’m sitting on the waitlist for now, because as someone who spends this amount of time on the internet, it’s worth at least a few dollars a month. Plus, I like the idea of paying money for several sites, rather than a pricey subscription that gets me access only to one website that may not be my main source of news (sorry, New York Times).

These days, passwords are still a primary mode of ensuring privacy and security on the web. As they’re a constant as we navigate across various websites, they start to acquire lives of their own. Ian Urbina delved into those secret lives in an interactive piece for the New York Times Magazine that includes a fascinating story of password recovery in the first days after 9/11. I recommend the videos that accompany the piece. Passwords have a unique power, being secret codes that we type dozens of times a day (or week). One man decided to use these secret codes as a form of self-help, to help himself forgive others, or quit smoking.

While there are numerous efforts afoot nowadays to do away with passwords, they’ve been around for ages. Another essay details the work that goes into outguessing passwords--and how easy it has gotten for many crooks, crackers, and phishers. He gives tips for creating a secure password, but surprisingly doesn’t mention using a password manager (I use LastPass).

And for our music this week, here are a couple songs by Mapei. She reminds me of Laura Mvula, and her song Believe is stunning.

You may have heard her song Don’t Wait, and she also has a remixed version featuring Chance the Rapper and the Social Experiment.

Thanks for reading!

Taylor Swift and Being Between Stars

Wed, 12 Nov 2014 16:00:00 +0000

Taylor Swift has been blowing up the music industry lately, first by surprising everyone with the beauty of her latest album. SNL dubbed it a result of Swiftamine, and I can certainly say I’m under the spell.

Then, pre-release, she removed her entire discography from Spotify. The Atlantic reflects on this decision by pointing out, “Owning music outright, instead of renting it through a streaming service, would be better for listeners and artists in the long run. Indeed, it would be better for just about everyone except Spotify.”

A founder of a service artists can use to self-distribute their music weighs in, and declares that “it’s amazing that the old record industry existed in the first place” by stating that the only reason musicians in the early music industry were successful was because “they took advantage of inefficiencies in the market”. iTunes and Spotify disrupted that market by offering sales of single tracks, and pay-to-play streaming formats, but “by removing herself from streaming services, Taylor Swift is intentionally adding inefficiency back into the market.” If the most efficient manner of gaining traction as an artist is to do everything yourself, then the fully-optimized, efficient music industry of today is where you want to be. Unless you’re Taylor Swift, you’ll probably also be broke.

A few years ago, Sasha Frere-Jones, music critic for the New Yorker, examined Spotify in the light of Thom Yorke’s decision to pull his music from the service.

“The issue beneath all the complaints about micropayments is fundamental: What are recordings now? Are they an artistic expression that musicians cannot be compensated for but will create simply out of need? Are they promotional tools? What seems clear is that streaming arrangements, like those made with Spotify, are institutionalizing a marginal role for the recordings that were once major income streams for working musicians”

Spotify aside, what powers T Swifts sudden dominance of the mainstream? First, it’s her true pop debut, but more importantly perhaps is the songwriting duo backing up most of her album. Aside from Swift herself, the two most-credited songwriters on most of her album are Max Martin and Shellback. Both Swedish, they are products of a unique culture in Sweden.

“In fact, Sweden is the largest exporter of pop music, per capita, in the world, and the third largest exporter of pop overall. And in recent years, the country has seized not just the message, but the medium as well: As the industry moves toward a distribution model that relies on streaming music services, the Stockholm-born Spotify is a dominant player, with 24 million users per month.”

As music takes over the mainstream and streaming services and social media dominate online music consumption and discussion, The Morning News revisits a conversation with MP3 bloggers. Music blogs were huge to me growing up, as I’ve discussed before in reaction to an in-depth examination of streaming, and the history of my own music involvement. The bloggers discuss the present and future of music–discourse and industry.

On digital music as competition for physical media, “Digital might have advantages in distribution and portability but the aura of the object is something digital can never replace. I think we’re clearly heading towards a music world where digital will be the indisputably dominant form but I can’t imagine one where physical music becomes anachronistic.”

The film Interstellar, directed by Christopher Nolan, has also been dominating discussions lately. I saw it this past weekend, and I’ve been thinking a lot about space ever since. The film is visually stunning, and it turns out that most of the space visuals are not only accurately rendered, but rendered for the first time visually using the special effects software used in this film (with renderers specifically written for the necessary scientific equations). Here’s a mashup of Interstellar along with other famous space films, like October Sky and 2001: A Space Odyssey.

The score of Interstellar is part of what helps make the film so powerful. Hans Zimmer scored the film, and as The Atlantic discusses,

“In the context of a movie that embraces the idea that emotions and feeling are the most important dimensions of human experience, the overwhelming nature of the score is all part of the way Interstellar works as a cohesive whole.”

Space, and spaceflight, involve the capability of looking back on earth from above. While we can see earth from above when flying in ordinary airplanes, being able to see the curvature of the earth has a serious effect on the pilots and astronauts that have experienced it. Referred to by some as the break-off effect, some freak out and have difficulty flying afterward, perhaps attributable as well to the relative isolation of their cockpits, as the article points out.

Interstellar travels to another universe, courtesy of a future NASA that has figured out how to send humans not only to Saturn, but beyond (through a wormhole). Meanwhile, in the current day and age, we’re still trying to figure out if we can send people to Mars. There is a company that is convinced that they can, and are raising money (and signing contracts for a reality TV series) to that effect. However, while the company, Mars One, has investments, applicants, and contracts, what they don’t have are technical specifications or capabilities for their mission.

I’ve been gone for awhile, but October was a hellishly busy month for pretty much everyone, it seems. Now that it’s November, I’ve returned! A lot more in the pipeline, about nerdy things like infrastructure (of high-frequency trading networks), container ships, lightbulbs, the effect of algorithms on photographs and memories, virtual reality, and so much more.

While I’ve been out of the newsletter world, I’ve been tweeting up a storm (as usual), and also wrote about my experience using MyFitnessPal and attempting to quantify my own health. I’ll probably post again about that toward the end of the year, as I amped up my self-tracking after that post, and have high hopes to go through some of that data alongside other measures. I now use a mood tracker, a period tracker, a habit tracker, and have been tracking the songs I listen to since 2007 (or earlier) using Last.fm. I think it’ll be cool, if I can get all the data out to play with, to see if there are any correlations. (Speaking of which, if you have experience working with APIs, let me know!)

Quantified Health and Software Apps

Thu, 16 Oct 2014 20:03:57 +0000

I went on a bit of a Twitter rant last night, about how MyFitnessPal doesn’t give me much helpful data:

Myfitnesspal is a good food-tracking app, but wish data were more helpful. (1/?)

— Sarah K Moir (@smorewithface) October 16, 2014

While it’s called MyFitnessPal, it doesn’t feel much like a pal, and feels more like a diet app than a fitness app:

Chides me when my foods go outside of goal ranges, but doesn’t use info about past meals or habits to make suggestions (2/?) — Sarah K Moir (@smorewithface) October 16, 2014

It’s like a friend congratulating you for eating a lot of whole wheat, but making a face because the egg you ate has a lot of cholesterol in it, even if it’s the only egg you’ve eaten that week.

Doesn’t track trends and offer recipes or workouts to combat them. Health and nutrition is subjective yes but I want more. (3/?) — Sarah K Moir (@smorewithface) October 16, 2014

The data it’s gathering, which for me is about four months of data, isn’t used to show any trends or to turn the data into more meaningful advice. As I point out, health and nutrition is subjective – while I may be focusing on eating more whole grains and vegetables, others may want to make sure they’re hitting a protein goal, or staying away from certain kinds of fat.

If I’m sharing all that data with an app, I want to get more out of it than a constant reminder that I eat too many carbs. (4/?) — Sarah K Moir (@smorewithface) October 16, 2014

There is a congratulatory and gentle chiding aspect to feedback, but nothing motivational. (5/?) — Sarah K Moir (@smorewithface) October 16, 2014

No, wow you’ve eaten veggies every day this week, keep it up! Or hmm you’ve been eating chocolate, try other foods to boost energy (6/?) — Sarah K Moir (@smorewithface) October 16, 2014

Congratulating trends that occur long-term, in addition to in-the-moment feedback, would be even more beneficial. Long-term trends are habits, after all. It’s great if I eat sweet potatoes because they have a lot of Vitamin A, but if they’re the only vegetable I’ve eaten that week, it could be beneficial to suggest other options. It could be a food that goes along with sweet potatoes, or a recipe that incorporates sweet potatoes in it, to encourage me to eat more variety yet maintain a healthful habit and food choice. Reward and help build habits, not day-to-day behavior.

(Not to mention there is no good way to track mood alongside nutrition but that’s another app, + privacy concerns) (7/?) — Sarah K Moir (@smorewithface) October 16, 2014

A bit of a detour from the expectations I have from a food-tracking app, tying together nutrition and mood tracking seems logical. Given the increasing evidence that ties together the microbiome of the gut and mood, as well as the role that nutrition plays in maintaining the health of that microbiome, it seems clear that mood tracking is the next step for apps to tackle.

(There are a lot of mood tracking apps out there (and I will be investigating them).)

There are clear privacy concerns in tracking your mood using a smartphone app, and the methods used to gather that data could be subjective, inconsistent, or raise more privacy concerns, but there is a benefit in exploring correlations between nutrition and mood. Mood tracking alongside nutritional trends is also beneficial alongside period tracking, but given that neither mood nor period tracking are included in Apple’s Health app, correlations can’t yet be drawn with this data.

Sad that most food tracking apps are diet and calorie based rather than nutrition and personal goal based (8/?) — Sarah K Moir (@smorewithface) October 16, 2014

It bugs me that the MyFitnessPal app revolves heavily around calories as a tracking metric. While it records nutritional information about protein and carbs, and those are visible in a different view of the app, the primary metric that is tracked is still calories. If someone wanted to create a diet or a nutritional goal that tracks protein or carbohydrate intake, a different view needs to be used, and the data is only shared in aggregate. I can look at a day’s nutrition to see where I’ve gone over my limit, but I can’t see which food had the saturated fat I wasn’t supposed to eat, or which food had the largest percentage of my carbohydrate intake. Nutrition data is available, but not in a valuable format.

Where do I go to track food intake for nutrition and health, rather than “fitness” and diet? (9/?) — Sarah K Moir (@smorewithface) October 16, 2014

I can see at a glance that the cheez-its I snacked on were almost 400 calories, but not that it’s the only source of bad fats in my diet for the day. The app is much more passive when it comes to nutrition. The ability to understand the food we eat, and the habits to change, relies on a better understanding of where the nutrition we eat is coming from.

I can make my own assumptions from the data, but there’s no narrative, and no mechanism for me to build my own outside of weight loss (10/?) — Sarah K Moir (@smorewithface) October 16, 2014

Without a diligent monitoring of the “nutrition” interface, and viewing my protein, fat, and carbohydrate intake as a series of pie charts (on a daily or weekly basis), there is no way to build an understanding of trends, let alone identify habits. There is no long view available.

If I eat more carbohydrates toward the end of the month, or tend to cycle between healthy weeks followed by more unhealthy weeks, I won’t know. The focus is less on health and nutrition, and defaults to a tracking of weight and caloric intake. Given that my weight has hovered in the same four pound range since I started tracking the app, a weight-based metric is not valuable to me. A nutrition-based metric, however, is.

Quantified self sads I guess. Too much data and little value wrought (11/11) — Sarah K Moir (@smorewithface) October 16, 2014

I added MyFitnessPal’s data to Apple’s Health app in an attempt to create this more meaningful narrative for myself. Liberate the data and be able to do something else with it outside the app. But so far, all I can do with the nutritional data is look at it as a series of line graphs.

I can’t even view the lines superimposed on the same graph to notice correlations. Not to mention that the data is displayed in grams; which is totally meaningless when it comes to gathering value from the information. These software apps act essentially as repositories, nothing more.

This challenge isn’t unique to these two apps. I discuss them because I use them, but they represent a greater issue—we have more data than we know what to do with, and lack the analytics and ability to turn this data into meaningful narratives for people. I can track calories by hand, but drawing conclusions about trends in my nutritional habits is something that software is better-suited to than I am. Make it easier to do the hard stuff, not easier to do the easy stuff.

Reading, Drones, and Georgie Washington

Fri, 26 Sep 2014 14:00:00 +0000

Americans are still reading books, Internet and all! Younger Americans are actually reading more than older generations, which could be partially due to the fact that with the rise of texting and social media, so much of our communication is text-based, so everyone is doing a lot more reading (and writing) in order to communicate with their friends. The original study is linked in that article and in this graph:

What are some other ways to get people to read books?

Well it helps a lot if your college library not only tells you the call numbers of the book, but it gives you precise directions to the location of the book, which is pretty awesome. Much more useful when navigating a giant library, like I have access to at the university I work at, as opposed to the smaller library at the university I actually attended.

Another good way is to send free books to the millions of soldiers at war. That’s a really great way to get people to read, and also lead to the rise of quality paperback books.

“By giving away the best it had to offer, the publishing industry created a vastly larger market for its wares. More importantly, it also democratized the pleasures of reading, making literature, poetry, and history available to all.”

Because people are still reading (and likely due to the trendiness of shopping local), independent bookstores are thriving, even as chains like Borders (R.I.P.) and Barnes and Noble start to fade in popularity.

Now it’s time for a

As I’ve mentioned before, everything is broken. On Wednesday, a vulnerability in bash was discovered, and while it was quickly patched, a similar-looking but different vulnerability was uncovered the next day, and that has yet to be patched. Bash is used in unix-based systems, most notably in Linux and Mac operating systems. Lots of server admins are panicking right now, but as consumers, we just need to sit and wait for patches (and if you use Linux, apply the first patch while you wait for the second). If you want to test your configuration, take a look at Brian Krebs’ rundown. I’ll be twiddling my thumbs until Apple releases an update.

Aside from gaping security vulnerabilities, I was recently fascinated by a couple essays [nerd alert]:

Why Google is Hurrying the Web to Kill SHA-1. A not-too-technical-but-still-technical essay about web security and why it’s semi-insecure and why Google is taking faster, unprecedented steps to make it more secure.
Inside Google’s Ambitious Plan to Change the Internet Forever. Starts with the premise of Google Fiber Internet, but largely discusses what would be possible if everyone had access to a “symmetric network” – equal upload and download speeds.
Software Version Numbers: A Neglected Opportunity to Improve Customer Experience. I have to write release notes as part of my job, so it heartens me to know that people do read them. Not to mention, this essay makes some great points about having trustworthy and reliable information included in version numbering. It can be somewhat random, and out of control (Google Chrome and Last.FM I’ve found to be the worst with the “version 32.0.324389404” numbering).

</Technical Interlude>

Emma Watson is leading a new UN initiative called #HeForShe, calling on men to do more as allies and pointing out what they have to gain by participating in feminism (hint: a lot). By treating gender more as a spectrum, men have just as much to gain as women.

Another realm where feminism is useful is economics. Many (developed) countries have aging populations, and are struggling to maintain their population growth because more women are choosing not to have children and the fertility rate is falling below the replacement rate of 2.1 (babies per woman). Sweden seems to have figured out a solution, however – offer paternity leave to the men. This mitigates the career hit that women take when they leave work to have a child, by allowing men to take time off from their career as well.

Something else that Emma Watson spoke to in her speech is that men don’t have room to be vulnerable or show anything but masculine toughness. This is clear on the web as well. Leah Reich wonders “if it all comes back to men. Not about banning or about even banishing them, but about the spaces men have carved out for themselves on the Internet, and the spaces they lack.”

One place it’s clear that women need to be more involved is in the tech industry. Apple is 70% male, and in the tech positions, that number rises to 80%. It’s never been clearer that that needs to improve than when people realized that the Healthkit app doesn’t track menstruation and fertility. For the record, Apple’s racial diversity is just as lacking. Also worth noting is that Google isn’t any better in either respect.

Drones are basically always in the news, at least the news that I read. DHL is going to use a parcelcopter (Paketkopter) to deliver packages to a remote island in Germany (DHL delivers most packages in Germany, in partnership with Deutsche Post). More background about the project is in DHL’s press release.

More promising, however, is the proposition to use drones in Africa to transport goods where infrastructure is lacking. The lack of infrastructure in many parts of Africa has meant that they are far ahead of us in some respects.

Apple Pay is revolutionary for many Americans, but the idea of paying for goods and services with your phone is already familiar to many Africans. Mobile payment systems such as M-Pesa in Kenya allows use of a mobile device to deposit, withdraw, or transfer money. It’s been around since 2007.

While mobile networks in Kenya served as a stand-in for banking infrastructure that didn’t really exist (or wasn’t easily accessible in more rural areas), that isn’t the case for banking infrastructure in America:

“In the U.S., where even the smallest towns have corner stores with ATMs, Apple Pay won’t have the radical impact that M-Pesa has had in Kenya. But the example of sub-Saharan Africa suggests the ways that mobile technology might eventually be able to displace developed nations’ complex physical infrastructure for finance and communication.”

Stars and Hey Rosetta! are going on tour together and I can’t go with my friends in Chicago but I might go see them closer to me. I haven’t been to a concert in a long time, so it might be worth it.

In honor of that, here is a new single from Stars called From the Night, and another single from Hey Rosetta! called Soft Offering (For the Oft Suffering).

Public Transit and Technology - Chicago Edition

Tue, 16 Sep 2014 18:24:28 +0000

The Chicago Tribune reports on a recent study completed by OECD on Metropolitan Governance of Transport and Land Use in Chicago. As the Tribune describes:

“The Chicago area’s transportation is hamstrung by a proliferation of local governments, the “irrational organizational structure” of the Regional Transportation Authority and the service boards and an antiquated formula by which transit agencies are funded, the report found.”

When reached for comment, the various transit organizations had no comment:

“Spokesmen for the RTA, Metra and Pace said officials had not read the 20-page report and had no comment. As it has previously, the CTA said last week that it opposes transit agency consolidation, as does Emanuel.

A superagency would be an unnecessary bureaucracy unaccountable to commuters that would divert dollars from train and bus service, said [CTA] spokesman Brian Steele.”

Per the Tribune, the report points out that:

““The current state of transit ridership in Chicago is relatively depressing,” concludes the report from the Organization for Economic Cooperation and Development, a Paris-based research agency whose backers include the world’s richest nations, among them the U.S.

The report found a lack of coordination among the four transit agencies and their four separate boards as well as insufficient accountability. Those issues intensify the economic impact of congestion on Chicago, estimated at over $6 billion in 2011 by the Texas Transportation Institute, the report said.”

Transit organizations in Chicago aren’t well-integrated, and leadership in Chicago opposes any integration or consolidation of those organizations. In the meantime, ridership is low and congestion (and its related economic impact) is high.

Contrast that with the recent article in Citylab about the importance of the smartphone in transportation.

“As more and more of the transport system falls into private hands and becomes fragmented, multi-modalism risks declining and cities will lose out on valuable data on where people want to go, how they travel, what’s slowing them down, and how the network is operating. A publicly-operated unified mobility app has enormous potential to eliminate barriers between modes, use existing infrastructure more efficiently, and bring the entire transport network to the smartphone.”

Privatized transportation systems, especially fragmented ones, means that cities lose valuable opportunities to find out more about their riders–and thus lose opportunities to attune their systems to the needs of their riders. For what it’s worth, Jason Prechtel writing for Gaper’s Block has closely followed the public-private partnerships that dominate Chicago public transportation. As the article continues:

“Better data about movement makes it easier for officials to site bike-share docks, or re-route buses to fit travel patterns, or add an extra train during rush-hour to meet demand. Instead of operating on a static schedule that forces users to adapt to it, a transportation network that’s monitored and adjusted in real-time can adapt to users. Just as the paved road launched a transportation revolution by enabling point-to-point travel via the car last century, networked technology can shift the paradigm again by making the user and infrastructure dynamic actors who respond to one another. This isn’t a trivial improvement—it’s a dramatic reimagining of how transportation systems operate.”

Transportation systems making use of ample data across the network have the ability to reshape themselves to meet the needs of customers–thereby reducing congestion, and increasing ridership.

“if U.S. cities can move past the fractured transportation landscape and embrace the challenge, their slow start isn’t necessarily a bad thing; it might even help officials avoid the mistakes of bad apps and refine the successes of good ones.”

Chicago has a long way to go before it can embrace and make use of technology across all of its public-private partnerships. Finding a way to integrate the data on ridership from Divvy, with the public transit usage stats from Ventra-carded services Pace and CTA, as well as Metra, could lead to some public transit innovation and some cost savings alongside transit improvements. Maybe claims of creating a “smart city” with “big data” could lead to some movement, but without improved partnerships and governance across transit organizations, Chicago’s public transportation situation seems destined to fester.

9/16/14 Update: Jason Prechtel wrote in an earlier Gapers Block column about the role of the RTA (regional transportation authority) which oversees the CTA, Pace, and Metra and which was responsible for uniting the three under one common payment system, Ventra. Prechtel on the RTA and Ventra:

“…both Gov. Quinn’s office and the SouthtownStar have called for finding ways to reduce waste and bureaucracy and eventually overhaul the entire regional transit system.

From this perspective, the need for a system like Ventra makes sense. Uniting transit fare payment under a single system is one major step towards merging the transit systems together under the RTA umbrella, and reducing overall transit costs and inter-agency squabbles.”

While that common payment system has been plagued with controversy and difficulties, perhaps the efforts of the RTA could lead to a unified transportation app for Chicagoans.

Three Types of Health

Fri, 12 Sep 2014 13:00:00 +0000

Public health in the U.S. tends to focus on chronic diseases (like cancer or diabetes), but in other parts of the world, much of the focus is on drugs that either no longer afflict the U.S., or aren’t cost-effective to treat.

Sickle cell anemia can be treated when it’s identified early. But that doesn’t happen much in the developing world, it is still a serious issue. So a diagnostic test that is simple, fast, and cheap is ideal, and currently in development.

Malaria isn’t a disease most Americans think of unless they’re going somewhere in Africa for a trip. A new diagnostic test (developed with technology that is also used in missile detectors) can diagnose malaria in four minutes in patients that don’t even show symptoms yet, and doesn’t even need a specialist to interpret the results.

Ebola is yet another disease that is more of an edge case–devastating, but rare, especially in the United States. For pharmaceutical companies, this means that it isn’t fiscally worth it to produce a treatment for ebola:

“When pharmaceutical companies are deciding where to direct their R. & D. money, they naturally assess the potential market for a drug candidate. That means that they have an incentive to target diseases that affect wealthier people (above all, people in the developed world), who can afford to pay a lot. They have an incentive to make drugs that many people will take. And they have an incentive to make drugs that people will take regularly for a long time—drugs like statins.”

Cancer is widespread across the globe, and has been around for millennia. For some kinds of cancer, however, genetic treatment is experimentally promising. Rather than attempting to destroy the cancerous cells, targeted treatments have been shown to cause cancerous cells to mature into non-cancerous cells.

Eat an apple a day, keep the doctor away… wear an Apple Watch, send your doctor your health data?

If you didn’t hear, Apple announced a bunch of new products. Most-hype among them was the Apple Watch, with lots of sensors to track your health. It remains to be seen whether anyone beyond Apple fanboys will buy them, but you may be tricked into wanting one.

The best thing I read about the watch is this analysis by Ben Thompson:

“I believe Apple’s long-term plan for Apple Watch is to own the wrist and to confer prestige and status with options like premium bands and 18-karat gold. To do that, though, they must compete not on technical merit but on the sort of intangible benefits that they always win with; chief among these is the user experience. A premium smart watch will win by yes, being fashionable, and yes, conferring status, but above all by doing a few things better than any other product on the market, and – this is critical – dispensing with everything else in the pursuit of simplicity.”

Aside from the watch, Apple announced two new phones– the iPhone 6 and the iPhone 6+. A current iPhone screen size (iPhone 5S) is 4 inches diagonally, while the iPhone 6 will be 4.7 inches and the 6+ will be 5.5 inches.

This means iPhones will gain a bigger screen, but with a relative cost for many:

“there are a lot of iPhone owners out there for whom — relative to the size of their hands — their iPhone is already bigger than the Galaxy Note was for the men who wrote those articles. It didn’t occur to those authors that their hands were probably larger than most women’s hands, and that the experience they had with the Note wasn’t altogether unlike how many women feel while using their iPhones today.”

Zeynep Tufekci wrote about large phones last November, making the case that larger phones are a product of gender-biased product design:

“Increasingly, on the latest versions of the kinds of phones I want to use, I cannot type one-handed. I cannot take a picture one-handed. I can barely scroll one-handed—not very well, though. I can’t unlock my phone one-handed. I can’t even turn on my phone one-handed as my fingers cannot securely wrap around the phone while I push a button with a finger.”

A darker form of health is mental health. Not much discussed, but for victims of domestic abuse, a vital element in their stories and their recoveries.

Football player Ray Rice was kicked out of the NFL after a video of him beating his (then-fiancee, now-wife) Janay Rice unconscious was released. After seeing or hearing about the video, many wondered why Janay Rice is married to him. Twitter decided to explain. Scores of domestic violence sufferers told their stories about #WhyIStayed, or in some cases, #WhyILeft. It’s powerful stuff, and if you are a survivor of domestic abuse, you may not want to read on.

A powerful TED talk from Leslie Morgan Steiner shares her very personal explanation for why domestic violence victims don’t leave.

When #YesAllWomen was trending on Twitter, Rachel Sklar shared her own story.

“And, crucially, it’s not just anger. It’s anger wrapped in fear, guilt, self-doubt, helplessness, sadness —so you shift from defensive mode into comfort mode, where you are the person who is calm and caring and reassuring. You’re the one who has it together. He’s the one who needs help, and you’re the one who’s helping. You learn to work around his triggers and do the things to soothe him and to leave parties quickly and to pick your battles.”

Kelly Sundberg wrote about her experience with beautiful language, but it didn’t make her story any less heart-wrenching to read.

“My counselor said, “You are taking everything he says, and playing it on repeat over and over again. You have to stop the tape.” But I couldn’t stop the tape. "

Jason Kottke linked to some more important links about why women stay with abusive partners: One anonymous account, another perspective from someone who worked at a battered women’s shelter, and a resource from the National Domestic Violence Hotline.

The most important fact that he pointed out was that “the National Domestic Violence Hotline has seen an 84 percent increase in phone calls” after the Ray Rice video was made public.

If you, like me, are feeling somewhat helpless in the face of all this suffering, you can do what I did and donate to the National Domestic Violence Hotline. According to the article about their phone volume, they didn’t have enough staff to answer all the calls they received even before this incident. They need all the help we can give them. Alternatively, you can donate to a battered women’s shelter near your home. If you don’t have the cash to spare to donate, do what you can to educate yourself about domestic violence – both how to recognize it, and how to help.

To make up for ending on such a somber note, here is HAERTS with their song Giving Up. Listen especially if you are a fan of HAIM or CHVRCHES.

Research, Atomkraft, and Mining

Thu, 11 Sep 2014 21:14:07 +0000

Hello! Long time no talk. I heard you all singing this song, and I’m back!

Here’s some strange and utterly fascinating research:

When a landfill was dug up to find out if Atari had buried its games there, it wasn’t a bunch of guys with a bulldozer. It was a team of archaeologists (and, of course, the media).
Weather models currently use data from the 1960s to the present, largely because not enough is digitized. But “nobody wants to spend the time and money it takes to scan and input 100 million pieces of pieces of old, musky, often disorganized paper.” Except, of course, for some very dedicated (unpaid) volunteers.
Spinach doesn’t have a lot of (absorbable by humans) iron in it. If you thought it did, blame a longstanding academic myth. This paper unravels the myth, but more interesting than the myth itself is the investigation into citation and source practices. I was transfixed with fascination reading this. I might be the only one that feels this way about citations.

Sometimes you get a job and realize later it’s not quite what you wanted and move on to something totally different. Sometimes when you do that, you were working in an atomic weapons depot then became a peace activist.

Your grandmother survived the bombing of Hiroshima. She was stigmatized by society, and traumatized for life. Turns out that kind of trauma is so powerful that it can cross generations.

Another kind of trauma, but one that can unite communities, is what has been happening in Ferguson. The media attention has died down, and many sites have quality coverage, but St. Louis journalist Sarah Kendzior is asking what happens after Ferguson?

Mining is also traumatizing, but to the earth.

As an accompaniment to this story about asbestos, here is a strangely humanizing story of an asbestos mine, told from the perspective of the mine.
These digitally altered photos draw a stark distinction between the destruction of a mine and the yield of the mine.
And if you read one thing in this newsletter, read this graphic novella about the mining industry and its effects on the environment. It’s by Kate Beaton, who writes Hark! A Vagrant. (Many thanks to Rob for pointing this link to me, many months after I read the story).

In case you were unaware (I was certainly unaware), ramen noodles are serious business. A company called Sun Noodles essentially dominates the American industry, and for good reason – they essentially created the market here. A great story about noodles, and the ultimate customer service. If you’re a sucker for “how it’s made” videos, you’re in luck. Here’s a peek into one of their factories.

And finally, in these days of living, loving, and transacting online, sometimes unlikely apps can become social media. Like a payment transaction app.

I’m going to leave you with a few music selections this week, to make up for lost time.

Jack + Eliza- Hold the Line. Listen because: They sound like Slow Club, or if The Shins had a girl.
The Prettiots - Boys I Dated in High School. Listen because: You miss the ‘50s, you want to listen to The Rosenbergs on a calmer day, you’ve heard of Billy Zygote.
Alt-j - Every Other Freckle. Listen because: You want to groove along to this smooth gravelly love song. The phrase “I want to be every button you press”. (Thanks to Lisa for sharing this song)
Hey Rosetta! - Kintsukuroi. Listen because: They sound like Bombay Bicycle Club, the phrases “she stood akimbo” and “raised on pixels”, anthemic indie rock.

Power to the Microbes!

Sat, 06 Sep 2014 21:02:26 +0000

Modern Farmer has an article about the “next green revolution”:

“A lot of materials used in corporate agriculture have the capacity to enhance plant growth and performance, but they suppress soil biology,” he says. The scorched-earth tactics he’d employed with his pesticides and herbicides, he realized, had worked all too well. The microbial life critical to healthy soils had become collateral damage. Afterwards, in a best-case scenario, Kempf could coax his cantaloupes and other crops to acceptable yields only by practically drowning them in fertilizer. He threw this approach out the window. Instead, by focusing on creating healthy soils, he’d let plants do what plants have evolved to do best when they’re given a fighting chance: grow like crazy.

By focusing on the base biology of his farm–the soil–this farmer, John Kempf, could increase yields and the health of the crops. Essentially, with healthy soil, you can grow hardier, stronger plants. As he determined, wholesale applications of pesticides would lead to temporary successes against pests that attack his plants, and copious amounts of fertilizer could coax the plants into growing larger and better, but once he turned his attention to the biology of the soil that his plants were growing in, the plants grew better than ever before.

Essentially, with healthy microbial life in soil, plants can grow bigger and stronger.

Science published an article about a bacteria in the digestive tract that helps to prevent food allergies:

A class of bacteria commonly found in the guts of people—and rodents—appears to keep mice safe from food allergies, a study suggests. The same bacteria are among those reduced by antibiotic use in early childhood. The research fits neatly into an emerging paradigm that helps explain a recent alarming increase in food allergies and other conditions, such as obesity and autoimmune disease, and hints at strategies to reverse the trend.

By focusing on the base biology of humans–the gut–this scientist, Cathryn Nagler, and her research team have found a way to prevent food allergies from developing. With a healthy gut microbiome, your body is better equipped to stay healthy and fight off diseases. While antibiotics kill bacteria that cause diseases, they also kill good bacteria that keep us healthy. This research is another step toward recognizing that people can better control and stabilize their health by cultivating a healthier gut.

Healthy soil leads to strong, healthy plants. Healthy gut leads to strong, healthy humans. Power to the microbes!

Parks, America, and Reading

Sun, 17 Aug 2014 22:42:01 +0000

It’s Thursday! Not Friday. Go to work tomorrow. When you don’t have to work, though, you can go outdoors! Because July is Park and Recreation month. So. If you’re not working, and it is nice outside, go outside. Weekend, planned. Just for you.

National parks are a great place to go outdoors. The National Parks Conservation Association is taking care to recognize people who identify as LGBT+ by doing more to preserve historical locations important to the legacy of LGBT+ life in the United States. More national parks, more important history preserved, lives validated. Recognized.

Mostly white people visit national parks. Fact. As of 2011, “only 7 percent of visitors to the parks system were black.” It hasn’t always been this way. The Boston Globe interviews geographer Carolyn Finney, who is recapturing the role of African-Americans in the history of the national parks system and the environmentalist movement.

Going outdoors isn’t the only way to interact with the outdoors! Oh no. We have the Internet. Which means we have webcams. Which means we have livestreams of animal habitats!

We can watch bear cams. Watch bears killing fish. Killing other bears, or witness the evidence of bear-killing unseen.

We can watch eagle cams! Watch eagles accidentally step on their babies. Demand that the poor runt who looks like it will die be rescued and helped! It dies anyway. Nature is cruel. Webcams make the cruelty of nature more visible.

(Those are not links to the webcams. Those are links to articles about the webcams that may or may not also include links to the webcams. By which I mean that they most certainly do. Feast your eyes upon the depravity and beauty of nature. Try not to get emotionally involved.)

If you do get emotionally involved, go to a zoo. Stare at the animals. Wonder how they’re feeling, what they’re thinking, why they do what they do. Make a career out of it. Tear up at the sight of black panthers in despair. Contemplate becoming a vegetarian, and wonder if you’d manage to eat enough protein or food without meat. Remember that barbecue exists.

Think about fish. Think about where fish come from. Realize that America controls the most ocean of any other nation. Think about how astonishing that is. Recognize that the United States imports the majority of its fish and seafood from other countries. The United States is still catching fish. One third of that fish and seafood is exported. Realize that globalization has “radically disconnected us from our seafood supply.”Cry into your imported scallops. Dream of real, American scallops.

Realize that America isn’t the best at everything. Such as education. Realize that Europe, especially Scandinavia, has education pretty well figured out. Go to Finland. Teach. Push your fifth-graders to work several hours in a row. Make them miserable. Realize that a 15 minute break makes them excited to learn! Wonder why America doesn’t do the same in their schools. Wonder why America doesn’t do the same in their offices. Go to the bathroom on a different floor. Pretend that detour is adult recess. Your imagination is not vibrant enough to turn stall doors into monkey bars. The toilet is not a swing. Look at your phone instead.

Maybe you’re not American. Maybe you’re Australian, teaching undergraduate students in the United States. Be confused by how dependent your students are on you, how much they value grades. Learn how to talk slower so they can understand your accent. Teach world history. Attempt to craft a mindset that does not include America as the center of said world. Use lots of maps to remind them of this. Become a better teacher.

Teach people what it means to care about reading. Support indie bookstores, but don’t antagonize people who buy books on Amazon. Instead, be thankful that you live in a town with an indie bookstore, because not everybody does. Realize you are in a privileged minority.

Teach people what it means to care about reading. Tell someone to read the Great Gatsby, because Jay Gatsby is a gangster. Reframe stories in a way that your readers can relate to. Become a better librarian.

Teach people what it means to care about reading. Don’t judge the number of books that they have read. Be grateful that they have read. Recognize that all reading is good reading.

Turn every link into a giflink. Listen to the oldies station on the radio. Call your significant other. Thank them, just like Sam and Dave tell you to. Make it to Friday.

Reading, Writing, and GIFs

Sun, 17 Aug 2014 22:38:46 +0000

The world is depressing. Let’s not talk about the plane that was shot down over Ukraine, or the Israel-Palestine conflict, or ISIS, or climate change. (those are all links to vox explainers I haven’t read, so beware maybe). Let’s instead talk about reading, writing, and gifs!

Read books. Read as many books as you want. Read as few books as you want. Read. And remember: “If someone reads fewer books than you do, it does not make them less intelligent than you. It does not even make them a worse reader. If someone reads different types of books than you do, it doesn’t make them a bad reader, either. It just means they are a different human being than you.”

Read books. Read children’s books! Remember them forever. Because “There are lots of books we can enjoy or admire, books that make us think and grieve and rejoice. But even those don’t necessarily belong to us as readers. No book belongs to you like a book you read when you’re young.” Read Harold and the Purple Crayon as a young child. Read The 13 Clocks when you get a bit older. Read Ella Enchanted when you are older still. Read them all over again when you are older to remember what it was like.

Read because of Reading Rainbow! Realize Reading Rainbow is now a for-profit app on tablets. Question whether an app to teach reading is more effective than just reading. “It’s easy to measure a kid’s understanding of basic math concepts through games: solve an equation to get the frog to leap to the lily pad, and congratulations, you can add! But literacy requires not just a set of discrete skills—spelling, grammar—but also a kind of comprehension that is hard to measure with an app.”

Read anyway. Read online. Read online so often that you want more from the medium. Explore interactive explanations of what you’re reading, explore it so far that you create and suggest alternatives. Have a dialogue with the text. Understand that not everyone can read online. Understand that even if they can read online, your fancy site may not even load.

Write online. Write by hand. Feel privilege in the choice, because “only 40% of the world’s population has access to digital technology” so there is no discussion about the obsolescence of writing by hand or teaching handwriting in schools. Understand how digital technology has changed how literature is written, that “Writing with electronic devices has affected structure, research and editing. It has affected order and it has affected rhythm, from the tap-tap-swipe of a typewriter to the swipe-swipe-tap of tablet.”

Unplug. Read more. Read more books. Revel in the Amazonification of the book marketplace. Visit France. Realize there are bookstores everywhere. Wonder why, and learn they have fixed book prices by law. Be happy you now have a reason to like France. Realize other European countries have similar laws. Get excited. Visit Europe.

Really though, visit Europe. Abandon your newsletter for the next two weeks. Go to Germany. Hike some cliffs. Walk a lot. Pack a lot. Relax a lot. Elude jet lag. Leave your laptop at home. Bring four books. Hope you have enough books.

Subscribe to other newsletters in the meantime to fill the void. Learn 5 intriguing things a day. Follow a newsletter that is currently on hiatus because you want to be surprised by its greatness when it returns. Learn about the Internet (and other things) from an Indian perspective. Get great longreads and an adorable giraffe illustration. Read up on the latest OSHA violations in the United States.

Go to Germany. Visit historical sites. Look at Hitler practicing his speeches. Revel in the raw humanity. Realize that evil people are people, not monsters. Wonder how you became who you are. Wonder how evil people become evil.

Look at more gifs to quell the fear and existentialism.

Meet Daryl Alexsy, who lives in Berlin but is originally from Michigan. She animates her illustrations, does lots of pen and ink drawings, and makes prints too. All of these gifs are belong to her. Click through to the original sources. Listen to soul music. Make The Supremes Pandora your soundtrack today.

Say goodbye for a little while.

Homemade Döner Kebab

Sat, 16 Aug 2014 22:37:24 +0000

Earlier this summer I vacationed in Germany and Switzerland. This was my third time in Germany, and my absolute favorite thing to eat there is Döner Kebab – a Turkish-German fast food that is flat bread, shaved lamb/turkey/chicken from a vertical spit, lettuce, tomato, onion, sometimes red or white cabbage, and a white sauce, and sometimes with a spicy red sauce or just red pepper flakes on top.

It. Is. Delicious. And you can pretend it’s healthy, since there are vegetables in it and the meat is cooking vertically, so much of the fat roasts out. I set out today to make my own version. I had to collect recipes from all over the place, so none of these recipes are mine (but I have linked to their sources).

First, I made the bread. There are two kinds of Döner, a Döner sandwich on a flatbread, and Dürum Döner, which looks more like a burrito. I wanted to eat the flatbread, and rather than try to find it, I decided to make my own…

Döner Kebab Bread

Ingredients

2 t (1 packet) dried yeast
1/2 t sugar
1 C lukewarm water
3 C all purpose flour
1 t salt
2 T olive oil
2 T plain yogurt
Extra flour, to dust
Extra olive oil, to grease
1 egg, lightly whisked
2 tsp sesame seeds

Steps:

Combine yeast, sugar and 2 T of the water in a small bowl, and stir until yeast dissolves. Set aside in a warm place for 10 minutes or until frothy.
In a large bowl, sift flour and salt together. If you don’t have a flour sifter, you can just whisk them together.
Make a well in the center of the flour and salt, and add the rest of the water, the oil, the yogurt, and the yeast mixture that you set aside.
Stir together with a wooden spoon until combined, then use your hands to bring the dough together in the bowl.
Put the dough on a lightly floured surface and knead it for 10 minutes or until smooth and elastic.
Lightly grease the inside of a large bowl with oil, then place the dough in the bowl and turn it to coat it with oil. Cover with a towel or plastic wrap, and set aside in a warm place for 2 hours (or until dough has doubled in size).

Two hours later…

]

Preheat oven to 425°F.
Punch down the centre of the dough with your fist. Turn onto a lightly floured surface and knead for 2-3 minutes or until dough has returned to its original size.
Divide the dough into 2 equal portions, and shape each portion into 7 inch by 12 inch rectangle.
Place on 2 non-stick baking trays (or 2 cookie sheets that have been sprayed with cooking spray) and press with fingers to indent surface. Brush with egg and sprinkle with sesame seeds.
Loosely cover the dough with plastic wrap and place in a warm place to rest for 20 minutes or until dough has risen half an inch or an inch.
Shape the dough into rectangles (or as close as you can get…), then cover with plastic wrap and let rise a bit longer…
Bake in preheated oven for 20 minutes or until the bread is golden brown and sounds hollow when tapped on the base. I baked it for 7 minutes, then swapped the cookie sheets on the racks, then cooked it for another 7 minutes. I then lowered the oven temperature down to 350° to prep for the meat, and put the bread back in for 3-5 minutes.
Remove from oven and place on a wire rack to cool.

So, there’s your bread! You have a nice 2 hour break in the middle of that preparation that is perfect to get the vegetables and the meat ready.

The vegetables…

Shred the lettuce. I got a head of iceberg lettuce and chopped it myself, but they also sell it pre-chopped.
Slice the tomatoes. I also de-seeded them because tomato seeds are gross.
Slice the onions. I try to slice them as small as possible because they can taste quite bitter.
Prepare the red cabbage! I prefer red cabbage to white, but it isn’t served completely raw.
1. Shred the cabbage as finely as you can. I tried to also remove as much of the white rib as I could.
2. Wash it until the water runs clear. I suggest being sure to do this step, as I hardly made an effort and my cabbage was still pretty bitter.
3. Add salt and pepper to taste, and then equal parts fresh lemon juice and olive oil. I used 1 T of each with about 1/3 of the head of cabbage (saving the rest for another dish).
4. Mix together!

You can also prep the vegetables beforehand. I shredded the cabbage and lettuce the day before to save time. Next, prepare the meat. I bought 1 lb of ground lamb at the store, but will probably mix it with turkey next time — the smell and taste of the lamb was very overpowering. The same forum thread with the bread recipe also had a meat recipe, so I used that.

Döner Kebab meat

Ingredients:

1 lb ground lamb (or equal parts ground lamb and ground turkey)
1 t salt
3/4 t ground black pepper
1 t flour
1 1/2 t ground oregano
3/4 t italian seasoning
3/4 t garlic powder
3/4 t onion powder
3/4 t dried mustard
1/2 t cayenne (if you don’t like spicy food, you can put in less)
1/4 t paprika

Steps:

Mix together all spices, then add to meat. I kneaded it like meatloaf, ensuring that all the spices were evenly distributed throughout. If you have a mixer, the recipe recommends that you “put all ingredients into a mixer with a dough hook and mix for 30 minutes.” The key is to get as much air as possible out of the meat.
Then, make a tightly packed loaf and put it in a greased loaf pan. If you have an insert that allows it to sit above base (so it doesn’t sit in its own grease), use it.
After mixing the lamb and spices, put the meat in a loaf pan.
Bake the meat at 350°F for 45 minutes to an hour, and turn it over halfway through to ensure even browning.
When the meat is done cooking, take it out and raise the oven temperature to 400°F.
Then, slice the meat as thinly as possible. Since mine roasted in its own grease, I put the slices down on a broiler pan and cooked the meat an extra 5 minutes to try to get more of a crust on it and roast out some of the grease. A higher temperature and a longer time would have probably helped here, but I was getting impatient!

So, there is meat, there is bread, there are vegetables, now you just need sauce! My sauce did not work out the way I wanted it to. I made a tzatziki-like sauce, following this recipe for Turkish Kebabs on AllRecipes and using half sour cream and half yogurt (I used Kroger brand plain greek yogurt). It was terrible, because it was far too sweet. I remade it with less dill and only sour cream, so I’ll be trying that next.

Mixing the tzatziki sauce! Clearly I went overboard with the dill.

Tzatziki sauce

Ingredients

8 ounces (1 C) sour cream
2 T olive oil
1 T lemon juice
1/2 t salt
1/2 t ground black pepper
1 T chopped fresh dill
1 clove garlic, crushed

(I used 1/2 t minced garlic from a jar)

Steps

Mix to combine all ingredients.

There were also recipes for yogurt garlic sauces, which was the same except without the dill, pepper, olive oil, and using yogurt instead of sour cream. I wouldn’t recommend doing this in the U.S. unless you’ve made your own yogurt or can find sour enough yogurt, because American yogurt is so sweet. I also liked that sour cream has a much lower sugar (aka lactose) content.

Bread sliced, toasted, and all ingredients standing by ready to fill.

It looks so authentic!

So it was a success, sauce aside. I’ll be trying various sauce combinations over the next few days as I finish up my leftovers!

What to Do, Messaging, and Ability

Mon, 23 Jun 2014 12:17:31 +0000

I completely forgot to send a newsletter last week, and for that I’m sorry. It was sunny, so I was actually outdoors. It has rained this entire week, so I have tabs for you now. Now, this evening, because I watched Return of the Jedi instead of writing this yesterday. So there’s that.

On to important things…

[Don’t do what you love](broken link removed). Instead, [do what you do](broken link removed). Better yet, [do what you hate not doing](broken link removed). If all else fails, do something, at least.

Need something to do? Send emoji to people. There are [new emoji](broken link removed). They hardly show up in my browser so I haven’t even looked at them much. Everyone keeps talking about this emoji.

The New York Times wrote about messaging apps like Line and KakaoTalk a few weeks ago. They use stickers instead of typical emoji. They also let you message and make phone calls for free over wi-fi. So there’s that extra bonus. Emoji just don’t cut it after you’ve used these stickers. Of course, the NYT doesn’t seem to realize that Americans DO want to use stickers and emoji, they just maybe haven’t been exposed to these delightfully excellent apps. For proof, just take a look at the proliferation of reaction gifs on tumblr. Conversations using images (that aren’t your face, Snapchat) are pretty excellent. This leads me to believe that the Americans interviewed by the NYT are stodgy businessmen or the like who are out of touch with how “the youngs” communicate.

If you’re colorblind, you could get some glasses that restore that colored vision to you, but it doesn’t really matter too much because [everyone sees things differently anyway](broken link removed). (Unless of course, you like knowing the difference between red and green lights).

It can be difficult to determine whether or not to use assistive technologies, to get rid of or correct something that “ordinary” society deems a fault (colorblindness, nearsightedness, deafness, etc.). No matter the ability, there is a certain commonality, a shared culture among people with similar inabilities. As someone who is nearsighted, I feel kindred to other people who wake up and see only blurs until they put on glasses.

Somewhat similarly, deaf people have a very real culture of deafness, one that may be difficult to participate in if you use a cochlear implant to regain some semblance of hearing. As that essay details, it can be a very tough decision, especially if you have hearing parents, to determine whether or not to use a cochlear implant or to learn sign language and embrace deaf culture–or attempt to do both.

Stanford’s alumni magazine had a feature last year by an alum who is deaf but can lipread well. She details the hazards of lipreading—accents, people who mumble, people who talk from different parts of their mouths—but also about the capabilities that it gives her.

“When I lipread, I leave the clarity of sign language behind. I attempt to communicate with hearing people on their terms, with no expectation that they will return the favor. The standards I am striving for seem ridiculous: I am trying singlehandedly to cross the chasm of disability. Might not my stubbornness be of more harm than good?”

A blind man wrote for the New York Times about not using a cane, and then deciding to use one and accepting that it “marked” him as a blind man.

And for a final note about why accessibility really is for everyone, if you are hosting or sponsoring a conference, it should have real-time captioning(link updated). Really, live-tweeting would be so much easier if I could be sure I got the quote right.

I’ve been spending the past few years discovering what I call groove/soul but is probably actually R&B (I’m really not good with genres). Ella Eyre with her song If I Go.

Listen if you like: Laura Mvula, Seinabo Sey, Naomi Pilgrim, Phoria, Erik Hassle, Rhye, Miguel, Frank Ocean, John Legend…

Further recommendations welcome!

This week’s newsletter was a little short, even though maybe it should have been extra long given the delay. It’s summer though. So if you’re curious what I read during the week and don’t want any quips alongside it, follow me on Twitter or on [Reading.am](broken link removed), where I’ve been somewhat obsessively collecting what I read. Bookmarklets are the best thing to happen to my tabs. That and the OneTab browser extension.

Children, Espionage, and Pain

Fri, 06 Jun 2014 15:00:47 +0000

Here’s what was important this week…

If you have a child abroad, and you’re a U.S. citizen, you can get your child U.S. citizenship as well. However, as Tori Marlan investigates “the rules that determine what babies can become citizens seem to be butting up against the modern circumstances under which Americans are having babies.” Most notably, modern practices that involve children being born before marriage, or through fertility treatments, or to same-sex couples. Proof is needed to prove that the child is “born of” the U.S. citizen–implied in heterosexual couples, but not as much for homosexual couples. Alexis Madrigal explores further implications that the future of reproductive technology will have for how we define parenthood.

In an unprecedented move, the U.S. indicted 5 members of the Chinese military on economic espionage charges.

As security expert Bruce Schneier points out, we’re guilty of nearly the same:

“The only difference between the U.S. and China’s actions is that the U.S. doesn’t engage in direct industrial espionage. That is, we don’t steal secrets from Chinese companies and pass them directly to U.S. competitors. But we do engage in economic espionage; we steal secrets from Chinese companies for an advantage in government trade negotiations, which directly benefits U.S. competitors. We might think this difference is important, but other countries are not as as impressed with our nuance.”

And James Surowiecki writes for the New Yorker on the fact that the U.S. got its start as an industrial power by engaging in just that kind of espionage, to the point where “State governments financed the importation of smuggled machines.”

As Surowiecki concludes,

“engaging in economic espionage is something developing countries do. When you’re not yet generating a lot of intellectual property on your own, you imitate. These days, China is going to try to steal, and the West is going to try to stop it. But the tactic of using piracy to leapfrog ahead? That looks like an idea it stole from us.”

As Spring comes to an end, the goslings I’ve seen all over town are turning into teenagers and will soon be full birds! It’s easy to forget that we’re surrounded by animals when you live in a city (except when the sidewalk is coated in goose poop). For those with even fewer parks, it’s easy to forget that cities bring with them their own biosphere of animals—gray squirrels and pigeons especially. An appreciator of squirrels, Jamie Allen, alleges:

“We city people have lost our connection to wild animals. Our pavement paradise, our automobile enclaves, and the pervasive technologies that sap our powers of observation have blinded us to our earthly neighbors. But nature is vibrating, even in the cities. Look in the skies. The birds are engaged in mortal combat. Coyotes and bears lurk on the outskirts. And in the matrix of tree canopy, squirrels are the cursor, the movement that gives voice to Mother’s existence here, even still, and always, long after we’re gone.”

As you’ve probably picked up on by now, I am a word nerd. Writing as a job makes me especially aware of the words that I use, the nuance between having a process cause a user frustration or pain or irritation. The words used when discussing health issues is even more important. As SARS and MERS and other global epidemics arise, how we communicate these diseases across the cultures of the world, and the rhetoric that we use, is increasingly important.

Not only that, but language is very important when discussing pain. Pain is increasingly clinicalized and medicalized–attaching a rating from 1-10 to pain felt, rather than asking for it to be described in words. While a pain scale may seem to create “quantitative” assessments, the nuance of the pain can be lost–and so can the human capacity for sympathy and understanding of the pain. This was somewhat brought about by the use of anesthetics for the first time:

“Anaesthetics rendered patients passive, unconscious bodies, stripped of sensibility, agency and, critically, words.”

(given that, it makes sense that many people fear waking up mid-surgery feeling pain but unable to speak out to stop it).

Social media is allowing us to recapture the descriptive aspects of pain once more (ignored since the 19th century), as a study is using Twitter to determine what words people commonly use to describe their migraine pain as it is happening. As one of the researching doctors points out, “The more you connect with your patient, the better you can treat them.”

In celebration of word nerddom, here is 43 minutes of every word in Star Wars in an alphabetical supercut of each instance of the word. Darth Vader makes a lot of plans.

If that really isn’t of interest, here is a song called Giants by the band Bear Hands.

Misogyny, Maya Angelou, and Words

Fri, 30 May 2014 20:40:56 +0000

A lot has happened since last week. As a heads up, the first portion of this post is about misogyny and the UCSB shootings last weekend. If you’d rather not read about it, skip below the comic!

Last weekend, a man murdered 6 people and injured 13 more. Misogyny is largely being credited (not much in mainstream media, however) as the primary driver behind his violence. The killer left behind several youtube videos and more than a hundred pages of a violent manifesto. His parents had reached out to his therapist, and the police met with him, but nothing came of the meeting. Part of this is because they based their judgment of him on their face-to-face interaction, rather than on his digital droppings of his thoughts and opinions, perhaps a misprioritization in our current world.

As I’ve written before, there is a real risk in defining people based solely on what they post on social media. But when so much of someone’s thoughts and feelings are revealed online, their narrative becomes more transparent. This man’s narrative was one of violent, extremist misogyny.

As Laurie Penny describes it for New Statesman:

““The ideology behind these attacks - and there is ideology - is simple. Women owe men. Women, as a class, as a sex, owe men sex, love, attention, “adoration”, in Rodger’s words. We owe them respect and obedience, and our refusal to give it to them is to blame for their anger, their violence - stupid sluts get what they deserve. Most of all, there is an overpowering sense of rage and entitlement: the conviction that men have been denied a birthright of easy power.”

Kevin Powell writes for CNN, calling for a broader definition of masculinity:

“Sooner rather than later we must ask ourselves when and how we are going to redefine manhood away from violence, retribution, guns and killing? When will we teach men and boys that power comes not from the barrel of a gun, that there are other ways to express or deal with pain or trauma, ways rooted in peace, love, nonviolence?”

Arthur Chu (recently internet-famous for his success and strategy on Jeopardy) takes nerd culture to task for being particular enablers and mythologizers of misogynist ideology:

“the overall problem is one of a culture where instead of seeing women as, you know, people, protagonists of their own stories just like we are of ours, men are taught that women are things to “earn,” to “win.” That if we try hard enough and persist long enough, we’ll get the girl in the end. Like life is a video game and women, like money and status, are just part of the reward we get for doing well.”

He, like Powell, sees an issue with the American culture of masculinity:

“we live in an entitlement culture where guys think they need to be having sex with girls in order to be happy and fulfilled. That in a culture that constantly celebrates the narrative of guys trying hard, overcoming challenges, concocting clever ruses and automatically getting a woman thrown at them as a prize as a result, there will always be some guy who crosses the line into committing a violent crime to get what he “deserves,” or get vengeance for being denied it.”

If you’d like a taste of how widespread the impacts of misogyny are, take a look at #YesAllWomen on Twitter, or the When Women Refuse tumblr. The former is primarily personal recounts of harassment and misogyny experienced, and the latter compiles stories and news articles about what happens when women refuse men.

A blog post from a couple years ago does well to establish how simple it is for misogynists and serial sexual harassers to be almost implicitly accepted, like a missing stair:

“Have you ever been in a house that had something just egregiously wrong with it? Something massively unsafe and uncomfortable and against code, but everyone in the house had been there a long time and was used to it? “Oh yeah, I almost forgot to tell you, there’s a missing step on the unlit staircase with no railings. But it’s okay because we all just remember to jump over it."”

A more recent blog post tabulates an escalating list of sexual harassment behaviors. If you’ve ever gotten irritated because you can’t talk to a woman at a bar, or because someone misunderstood your intentions, this list is a great example of why. I am personally familiar with almost the entire list.

Wednesday morning, Nobel Laureate Maya Angelou passed away. The Paris Review interviewed her in 1990, and it’s a beautiful interview. On her life:

“Oh my God, I’ve lived a very simple life! You can say, Oh yes, at thirteen this happened to me and at fourteen . . . But those are facts. But the facts can obscure the truth, what it really felt like. Every human being has paid the earth to grow up. Most people don’t grow up. It’s too damn difficult. What happens is most people get older. That’s the truth of it. They honor their credit cards, they find parking spaces, they marry, they have the nerve to have children, but they don’t grow up. Not really. They get older. But to grow up costs the earth, the earth. It means you take responsibility for the time you take up, for the space you occupy. It’s serious business. And you find out what it costs us to love and to lose, to dare and to fail. And maybe even more, to succeed. What it costs, in truth. Not superficial costs—anybody can have that—I mean in truth. That’s what I write. What it really is like. I’m just telling a very simple story.“

Angelou also discussed at length her writing process and how she interacts with her writing:

“I really love language. I love it for what it does for us, how it allows us to explain the pain and the glory, the nuances and the delicacies of our existence. And then it allows us to laugh, allows us to show wit. Real wit is shown in language. We need language.”

Jason Somers would agree with Maya Angelou about language, especially the English language. So much so, that he declares in a blog post that you’re probably using the wrong dictionary:

“Google’s dictionary, the modern Merriam-Webster, the dictionary at dictionary.com: they’re all like this. They’re all a chore to read. There’s no play, no delight in the language. The definitions are these desiccated little husks of technocratic meaningese, as if a word were no more than its coordinates in semantic space.”

He contrasts these dictionaries, which focus on a basic definition, with an old version of Webster’s dictionary:

“Notice, too, how much less certain the Webster definition seems about itself, even though it’s more complete — as if to remind you that the word came first, that the word isn’t defined by its definition here, in this humble dictionary, that definitions grasp, tentatively, at words, but that what words really are is this haze and halo of associations and evocations, a little networked cloud of uses and contexts.”

Artist, author, and poet Austin Kleon feels the same way about dictionaries. Indeed, as I’ve mentioned in an earlier newsletter, definitions of words tend to be perceived as prescriptive, rather than descriptive. An essay by Catie Disabato disavows this notion, reminding us that:

“Words are symbols. They have no inherent meaning, only the meanings we give to them, collectively, as a group.”

Using a more descriptive dictionary could perhaps help reverse this trend. Digital enhancements and unexpected shifts, like moving from print to digital dictionaries like those embedded in our laptops and that exist on dictionary.com, have also shifted the definitions of words. Alexis Madrigal looks at how autonomous vehicles could shape how we refer to drivers, contrasting it with the definition of a computer:

“a computer was a human for more than two hundred years. And in the span of some decades, that meaning has been completely and totally drained from the word.”

Matt Buchanan explores this shift further by commenting on the creation of a glossary of terms for Twitter:

“Perhaps the recently erected Twitter glossary, designed to explain things like “retweet” and “favorite” to bewildered new users, is indeed helpful to someone (dads?). But to whatever extent it is a guide to what Twitter is, it’s also a glass-and-steel-condo-like monument to what the Internet was, when some words meant other things, like “favorite,” which was (n) a thing you liked more than any other thing, not (n) a hollow unit of social currency or (v) a thing that one does to remind another human that his or her life has some value to you that is greater than absolutely nothing.”

Enough nerding out about words, here is some music (full of WORDS!) words are beautiful (I lied about stopping nerding out). There is a band called Bad Bad Hats. Given their name, their hat taste may be questionable but their songstyling is most certainly not. Their song Super America is superb. It is free to download. The EP that it is a part of, is also free to download, and pretty great. You should download it and listen to it. If you like Bad Bad Hats (even as a fashion choice, I’ll forgive you), you may also enjoy Slow Club and their song Giving Up on Love.

If you enjoy words and dictionaries as much as I do, I will remind you of the Oxford Dictionaries blog, and also point out this book about the creation of the Oxford English Dictionary (I haven’t ever managed to get through it, however).

Memorials, Public Health, and Empathy

Fri, 23 May 2014 15:00:45 +0000

Here’s what was important this week…

The museum memorializing the victims of the September 11th terrorist attacks opened this week. Steve Kandell wrote about visiting it: The Worst Day of My Life is Now New York’s Hottest Tourist Attraction. The photographer and people-person-extraordinaire behind Humans of New York spoke to someone who also went through the museum. It opened early for survivors and family members of victims. Everyone agreed that the gift shop was in bad taste.

One of my favorite blog of odds and ends, Futility Closet, had two posts recently that speak to the difficulty of colonizing a country. When the Spanish conquered the West Indies, the conditions they imposed on the natives were so poor that they were committing suicide in great numbers. So great, in fact, that:

“In the end the Spaniards, faced with an embarrassing labor shortage, put a stop to the epidemic of suicides by persuading the Indians that they, too, would kill themselves in order to pursue them in the next world with even harsher cruelties.”

In Puritan New England, conditions were so bad for the conquerors (I mean settlers…) that children who were captured by Native Americans often didn’t want to come back. This was a somewhat popular theme for historical fiction novels like Calico Captive (which I read when I was younger). One possible reason for this, mentioned in the post, is that “The Mohawks were much more indulgent of children than the colonists, and women were counted equal to men and played an integral role in society and politics.”

Speaking of colonialism, Ta-Nehisi Coates has published a long essay making the case for reparations. It just came out Wednesday, so I’m saving it for this upcoming long weekend. You should too.

In unsurprising news, everything is broken. Especially everything having to do with computers. As Quinn Norton puts it so perfectly:

“There’s your choice: constantly risk clicking on dangerous malware, or live under an overpass, leaving notes on the lawn of your former house telling your children you love them and miss them.”

According to tech startup entrepreneurs, the way we do laundry is broken too. The snark that the author exudes is the perfect amount for this article.

When it comes to public health, malaria is still powerful and flourishing in Cambodia. The essay also points out that malaria used to be far more widespread than it used to be, but rich companies poured money into treatment and eradication, and now malaria is something Americans only think of when they plan a voluntourist trip to Africa and need to get a shot before they leave.

HIV is also very stigmatized, so even though it’s preventable, infection rates aren’t falling. The persistence of HIV across countries in Africa could also be explained by undiagnosed parasites, which cause sores in the vagina thereby increasing HIV transmission rates.

If you’d like to spend your Memorial Day weekend crying, read an essay about two peas in a pod becoming one after a woman’s best friend is diagnosed with cancer. Or you can read the essay about an oncologist whose wife is diagnosed with cancer. Or you can read both if you are in need of a really, really good cry.

I finally read Leslie Jamison’s essay collection The Empathy Exams, (the title essay of which I shared several weeks ago). Highly highly recommended, especially if you are in need of some introspective thought. Most of the essays are available online for free if you don’t want to spend the money on the book as well.

Sara Henderson, a writer on many topics around accessibility and ability, reflected on The Empathy Exams and empathy in doctors in general--especially giving them a well-rounded education to help with this. (My family doctor for awhile was actually a history major.)

An essay in The Rumpus looks at teaching medical students how to perform a gynecological exam on a woman, with their instructor taking care to note:

“Each time you touch a patient, you need to let her know beforehand. No surprises. It’s her body, and she’s granting you the privilege to be in her physical space.”

The author (and instructor) takes care to point out the importance of their role. Much like Jamison in her role as a medical actor,

“We are alive, awake, and very aware that they will learn from us tonight the privilege and responsibility of providing good medical care to women.”

As a last word this week, don’t use the term grammar nazi. Do listen to this song by The Hood Internet, mashing up Waka Flocka Flame and Twin Shadow. Primed for windows-down-summer listening.

Algorithms, Confidence, and Infrastructure

Thu, 22 May 2014 18:10:40 +0000

Every so often the Oxford English Dictionary adds new words. It adds them to its online dictionary with far more frequency than its physical tome, given that a physical dictionary is quite a bit more difficult to update. It [released a list of new words yesterday](broken link removed), and while a few are new words entirely (bikeable) others are new definitions of familiar words. The “tumblr definition” of ship is recognized (and [boy is the tumblr community excited about it](broken link removed)) and a definition of thing that accounts for the phrase “is that a thing?”

Daniel Temkin put together an Internet Directory with a scrolling and searchable list of all registered domains with a top level domain name ending in .com

Ted Striphas was interviewed about the effects of algorithms (such as the ones that define the order of google search results, or what shows up in your facebook newsfeed) on culture. As he puts it, “The issue may come down to how comfortable people are with these systems drilling down into our daily lives, and even becoming extensions of our bodies.”

Quinn Norton wrote a short story called Agency about a future where algorithms and and machine learning are fully ingrained into our lives, for better or for worse.

The European court of justice recently ruled that a man had the right to request that Google remove “personal material” that appeared prominently in the search results one got if you googled his name. Many are saying that this entitles users across the globe to “the right to be forgotten”, but as Viktor Mayer-Schönberger (author of Delete: The Virtue of Forgetting in the Digital Age) points out in an article for The Guardian:

“Such a deletion right has existed for 20 years, and very few of us have used it. There is little reason to believe that will change. Moreover, search engines don’t have to redesign themselves to comply. Google is already handling millions of deletion requests for copyright violations every month, so even a couple of hundred insisting individuals won’t make much of a difference.”

Natasha Lennard has more detail about the overall ruling in her Vice News piece, taking care to detail why the ability to control search results about yourself is valuable:

“Online lives are both accumulative and static. We seemingly leave the past behind us with the possibility of self-renewal and reformation. But online, many sets of online selves coexist contemporaneously and all at once.”

The website fallingfalling.com has an entrancing visualization of colors falling into one another, which, with the sound on, makes you realize how much hearing is tied to visual perception.

A few weeks ago The Atlantic published a long article about the confidence gap between men and women (which I have to admit, I didn’t read). What I did read was a bunch of response pieces or pieces revolving around the confidence of women. The perception of women has a lot to do with their confidence level, and how displayed confidence is perceived (something Jill Abramson, former editor of the New York Times, may have realized with her abrupt termination).

Writer Hila Shachar blogged about the different gender perceptions of seriousness, as part of a larger essay on being nice as a feminist:

“When men are serious, it’s sexy; it implies a commanding personality, someone who’s in control, someone with a backbone. When women are serious, they are bitches or unattractive, humourless hags, in need of sprucing up and “feminising”. How many times have I heard some of the cleverest women I know being called “feminist bitches” simply because they don’t conform to the “nice” girl image? Or, simply because they ask difficult questions and expect serious answers.”

Laurie Penny writes about how confidence and gender plays out on the Internet (shirtless torso at the top of the article, potentially nsfw) and how violence toward women affects their ability to be confident and “good”:

“Gendered violence, and the threat of gendered violence, has always about scaring women into submission. It’s a warning: Reach too high, walk too tall and bad things will happen to you. We are raised on tales of what happens to little girls who stay up past their bedtime, who go out alone, who read dangerous books and meet dangerous people and think dangerous thoughts. Good girls shouldn’t. Good girls don’t.”

Lastly, Ellen Chisa wrote about her own confidence issues, which manifested as a drive to “be excellent enough” in order to succeed (before realizing that her gender was holding her back):

“No one else thought there was a problem? Maybe there wasn’t a problem. Maybe I wasn’t as excellent as I thought. I was working as hard as possible, but it still seemed like something else was going on. Maybe I was afraid I couldn’t cut it, so I was inventing sexism that didn’t exist? So I decided I’d just try harder. Lean in. Be excellent.”

All the essays are worth reading, but the last one spoke to me the most.

The Daily Overview email newsletter (and website/instagram feed) provides a daily dose of gorgeous overhead view of a location. I’m a recent subscriber, and it’s great to take a moment in the clutter of the rest of the email to enjoy seeing a sliver of the world from a different perspective.

Perspective is important when examining the world we live in. Our worlds are engineered now more than they perhaps ever have been, with infrastructure and products supplying, surrounding, and supporting our lives. Julian Oliver wrote a Critical Engineering Manifesto about the effect that this engineering of everything has on how we interact with our world. Sam Hart interviewed him for VVVNT and it really is fascinating:

“So much of our environment is engineered: How we eat, how we move, how we remember, how we communicate are all tangibly and deeply affected (at least in the west), by engineered infrastructure, engineered principles, concepts, and ideas. So it follows, to not engage engineering on its own terms, or even have a basic vocabularic interface with it, is to be condemned or to limit your ability to become an actor and have a transformative effect.”

With such an engineered world, it’s perhaps almost expected that some things will be engineered with a bias. That bias, as often is the case, can be gender- or sex-based. Laurie Penny’s earlier article asks “Is the Internet Intrinsically Sexist?” due to the amount of violence endured by women online. Laura Mandanas writes for Autostraddle about whether or not inanimate objects can be sexist (spoilers, yes they can):

“Although women have been riding in automobiles for about as long as they’ve existed, car safety features have historically been designed and optimized for male-assigned bodies only.”

And now that you’re afraid of riding in your car (or afraid for your friends), you can be concerned about the effects of something uncommon now that could be more common in the future–virtual reality. danah boyd’s experience with Oculus Rift (the virtual reality headset recently bought by Facebook) made her throw up, and she wanted to figure out why. In a blog post expanding on a Quartz article, she goes into great detail on some research that she was involved with about the effect of sex hormones (such as androgen) on visual perception and spatial processing, and which kinds were emphasized by the brain depending on your biological chemistry. For those who questioned whether an object could be sexist, she clarifies:

“Sexism is prejudice or discrimination on the basis of sex (typically against women). For sexism to exist, there does not need to be an actor intending to discriminate. People, systems, and organizations can operate in sexist manners without realizing it. This is the basis of implicit or hidden biases. Addressing sexism starts by recognizing bias within systems and discrimination as a product of systems in society.”

Chvrches is a great band with some great songs. One of those songs is called The Mother We Share.

Wild Cub is another great band with great songs. They covered Chvrches’ song The Mother We Shared. It came out great (predictably). Give it a listen.

Thanks for reading!

Software, Sharing, and Music

Fri, 09 May 2014 18:12:24 +0000

Here’s what was important this week…

Software is everywhere lately. My boyfriend asked me what I thought the next big website would be (after the success of Google, Myspace, Facebook, Twitter, etc.), and I realized it’s just as likely (if not more likely) to be a software application rather than a website. Paul Ford took some time to enshrine some works of software in a “software canon” – Microsoft Office, Photoshop, Pacman, the Unix operating system, and eMacs (which I’d never heard of until this essay came out).

Software has had a noticeable effect on our day to day lives (especially those with smartphones), but it’s also had a huge impact on music and the way it’s created, recorded, and produced. Fact Magazine went through 14 works of software that shaped modern music (electronic music started way earlier than I thought). One of those software applications is Auto-Tune, and the Sounding Out! blog happened to post about the history of Auto-Tune.

The sharing economy has been a big buzzword of the last few years, driven by the rise in services like Air BnB, Lyft, Uber, and many others. Community-focused efforts like tool libraries and gardens are viable based on their inclusion in community networks. However, the more recent versions of the sharing economy involve making money off sharing something that one person has with another person that doesn’t. In some cases it’s based on the true utility of it, share something you’re not using with someone who will use it. It could be an exchange based on a developing trust in American society.

However it’s just as likely that it’s based on desperation. Growing as the nation recovers from the throes of recession and unemployment, the sharing economy provides a way for unemployed people to take advantage of assets that they still possess (a car, an apartment, a home) and monetize them into a form of income. And in terms of unemployment options, renting out your things can be far better than renting out yourself.

I’ve written here before about the importance of vaccinations. So let me write about that again. Communist Hungary fought several polio epidemics before managing to reach a high enough immunity among the population. Fascinating as well is how there were two separate types of vaccines involved in this effort, and both were vital to eradicating the disease.

And because everything in the world lately has to feel doomed, polio is not completely gone! In fact it’s coming back! And so are other diseases. At least we got rid of smallpox.

We may be doomed but at least there is still good music in the world. Some of it is on vinyl, and being listened to by a librarian, whose husband is the owner of a large vinyl collection. The librarian blogs about it, it is delightful, and Collectors Weekly interviewed the two of them. It is also delightful.

To add more delight to doomsday feelings, here is an indie-folkish song by The Family Crest called Love Don’t Go

(to continue tech-speak and tie 2 out of 3 topics together…).

The Internet of Things (another huge buzzword lately) is something that people talk about far more than it’s actually happening (but that’s probably just the tech crowd). As Tim O’Reilly points out, we’re still pretty far off from a full-fledged Internet of Things, but:

“there are many “halfway house” applications that are really Internet of Things applications in waiting, which use humans for one or more parts of the entire system. When you understand that the general pattern of #IoTH applications is not just sensor + network + actuator but various combinations of human + network + actuator or sensor + network, you will broaden the possibilities for interfaces and business models.”

We’re all just cogs in a machine. While the existence of Mechanical Turk as a service makes me sad for many reasons, it’s clear evidence that at least for the time being, humans are still better at enough tasks that there is an entire marketplace devoted to outsourcing those tasks to humans instead of machines. Plus there is still the entire workforce that indicates humans still do something. Luckily, as Mike Loukides writes on O’Reilly’s site, the Internet of Things isn’t going to be successful unless it ends up as interoperable as the Internet itself. If devices can’t communicate with each other, you don’t have an Internet of Things, you have Internet of Thing + Internet of Thing + Internet of Thing + Internet of Thing, each with their own app!

The economy still needs us, for now. Thanks for reading.

Journalism, Networks, and Grief

Fri, 02 May 2014 21:48:13 +0000

Here’s what was important this week….

Felix Salmon, a formers Reuters journalist, wrote a screed about why publishing news with the readers in mind is more valuable than breaking news.

As he puts it, “when journalists start caring about scoops and exclusives, that’s a clear sign that they’re publishing mainly for the benefit of other journalists, rather than for their readers. “

Even more clearly, and something that I can relate to easily, is the idea that:

“Readers come first, and all decent publications have their own readership: they shouldn’t be so meek as to assume that their readers will have invariably found the same news elsewhere, just because someone else’s version arrived a little earlier.”

When you spend most of your time on the Internet surrounded by, to borrow his phrase, media navel-gazers who lives on Twitter, everything starts to seem like unimportant, old news. But thankfully, when you talk to others outside of that arena, it is easy to remember that news that seems everywhere and overdone in one circle could be totally absent in another.

Of course, knowing those people is important in its own way. Journalists tend to have extensive networks, and networks are not only powerful for story generation, they help nurture good ideas.

“the fact that so many of us are writing — sharing our ideas, good and bad, for the world to see — has changed the way we think. Just as we now live in public, so do we think in public. And that is accelerating the creation of new ideas and the advancement of global knowledge.”

In fact just “the effort of communicating to someone else forces you to pay more attention and learn more.” and the more meaningful the audience, the better the result.

Networks underpin our worlds, and we need to organize them to help make sense of them. The tree of life has become more of a web of life, as our world becomes more connected. This RSA Animate video details why mapping these networks is so important.

Networks of people nurture good ideas, but networks of information can produce a similar result. Being a polymath, or someone who studies many different subjects, allows you to make connections and come up with new ideas by exposing you to different fields, and the more you learn throughout your life, the easier it is to keep learning more things.

And for today’s morbid moment, I ended up reading a lot about grief this week. The Ask Polly column at The Awl is consistently excellent, and this week she gave advice about handling the grief of an unexpectedly dead father:

“When you lose someone very close to you, someone who makes up this essential part of your history and your future, your worldview shifts dramatically. You have a palpable feeling that everything and anything good can disappear at any time.”

And in a personal essay, Peter Welch reflects on loss after losing two people in a short period, one very close to him and another related to his girlfriend.

“When I reflect on losing someone, I wonder how I’ll feel, since it’s different each time, and it makes my brain snap. I wonder how many people will feel sorry for me, and for how long. I wonder what sort of behavior I could get away with. Quit my job? Leave the state? If my whole family died, I could do pretty much anything, as long as I left a note. Less for one parent? A friend? The question of what to do after the hypothetical loss forces its way into my mind, blocking the horror of what could be lost before it hits me.”

In an interview with George R. R. Martin, he reminisces on the loss of his father and the use of reading as escapism:

“The night my father died, I was in Michigan and I got word from my mother. I couldn’t get to a plane until the next day, so I sat around thinking about my father, the good and the bad in our relationship. I remember I opened whatever book I was reading, and for a few hours, I was able to stop thinking about my father’s death. It was a relief.“

I’ve shared music from LAYLA before, but she just put out an EP on Soundcloud, and it’s really great.

Masculinity, AIM, Ads, and Cops

Fri, 25 Apr 2014 20:00:24 +0000

Here’s what was important this week…

I treated myself to ice cream last night (from the freezer, not a lonely ice cream shop date with myself) and it was delicious. While I gained weight from starting an office job after college, I still have the privilege of avoiding most body policing placed on women.

However, men suffer their own share of body policing. In Hollywood, this manifests itself as an obsession with fit bodies, and fitness. Mens Journal examines the issue, speaking mostly to trainers and talking about the pressure for actors to get “fit” in order to land coveted roles. It’s so important to the industry that:

“There are dozens of hormone-replacement clinics in and around Hollywood, and their business is booming. But there are significant risks: Hormone therapy accelerates all cell growth, whether healthy or malignant, and can encourage existing cancers, especially prostate cancers, to metastasize at terrifying rates. Testosterone supplements can lower sperm counts. For many, the risk is worth it.”

Fitness is just one aspect of a narrow set of masculinity standards imposed on men. For many men, high school is one of the more painful places that these standards are enforced. Well-documented in this great book by sociologist C.J. Pascoe, an essay in The Walrus gets to the heart of many of the standards. A new sex ed program in some Canadian schools works on teaching these high school boys not only aspects about sex that are often glossed over in traditional sex ed courses, it also focuses on relationships, gender identity and expression, and explores these things in a safe space. Importantly,

“Teaching young men to trust, communicate, negotiate, and empathize does not undermine or threaten their manliness. It expands their humanity. It reclaims men’s possibilities.”

Something else that helps men reclaim their possibilities is by supporting women, becoming advocates for them in the workplace, being feminists… Shanley, a writer on diversity in tech, wrote an essay about what men can do to help women if they are in a position of power (in her case, speaking directly to white men in tech). It’s a bit profanity-laden and not completely generalizable, but makes some great points.

Speaking of technology, do you remember when AIM was the greatest thing ever?? Because I do. While Gchat removes away messages and busy statuses, and our phones make us available at all hours of the day, the days of AIM recall the nostalgia of an away message that advertised “look at me, I have a life that is more important than the computer” rather than the daily photos of your vacation that you upload to Facebook.

AIM is 20 years old this year, which could explain why there are three essays about it this week. Mashable profiles the rise and progressive abandonment of AIM, from the AOL perspective. N+1 has an essay from an MSN engineer that worked on the rival service, MSN Messenger Service, that discusses some of the more basic things around programming in the nineties, as well as how he attempted to compete with AOL and integrate the services.

Both essays discuss a gradual bloating and bureaucracy-laden company culture that led to the downfall of the services, and as many large companies are overtaken as AOL and Microsoft have been by this culture, it will be interesting to see what happens next. While Facebook and Google have expanded (and seem to continue to do so), other companies are shedding services and employees along with earnings. Larger monopolistic entities like Microsoft may be able to break apart or work on a shift from inside the company toward a more innovative and iterative culture, but I’d expect that many which can’t, like AOL? or Yahoo? or large institutions like government or higher education, will stagnate instead.

With net neutrality potentially dead, there is good news for net neutrality activists, while advertising, higher costs, and throttled internet performance could be the future of the Internet. Apparently, it can get worse than it currently is. If you want to play with some data around Internet speed + cost across the world, the folks who run http://speedtest.net (the best place to test your connection, in my opinion) also make available the data returned from those (probably) millions of consumer tests. If net neutrality is dead, then the Internet will join the rest of America as a “meritocracy”.

The latest trick of advertisers is perhaps something you’ve expected or already noticed – native advertising. The sponsored posts in Instagram, Tumblr, and Pinterest, the Facebook News Feed posts recommending that you like a page, or the stories by Dell that insert themselves casually at the end of my Feedly RSS feed from time to time. While I’m somewhat okay with native advertising, as it can be high quality content from brands I care about, the more it infiltrates the web, the harder it could be to determine whether content is intended to sell a product or promote a service, or whether it is legitimate content from a website. Of course, throwing the word “legitimate” in there begs the question of whether website content can be assumed to be unbiased from brands and advertising influences already, but that is where conflict of interest disclosures come to play in journalism, at least. As native advertising matures, it becomes clear that for advertisers, “It’s no longer just your clicks they want, it’s your time and attention. Welcome to the Attention Web.”

If you watch this generic brand video, you’ll never have to watch another advertising video again because they’re all just like this one. (If you install AdBlockPlus, you really might never have to watch another advertising video…)(*not sponsored by AdBlockPlus).

Another tactic that brands are trying to engage consumers is social media outreach. Of course, depending on your brand, you may or may not want to invite consumers to the conversation. If you’re the NYPD, you probably don’t want to request that people tweet photos of themselves with members of the force, or you might end up with #myNYPD trending for all the wrong reasons. Of course, #myNYPD was not a total failure, as it got people talking about the role of the NYPD in their lives. As an essay in Brooklyn Magazine discusses, “the myth of a friendlier NYPD is only true for parts of New York residents. Maybe that will change, but it will require the NYPD to take a long, hard look at the photos posted on Twitter and acknowledge that those images are just as much a part of their identity as any others.”

As a reward for making it to the end of this post, here is a DJ mix from Kristian Nairn, who is a great DJ and also happens to be the actor that plays Hodor on Game of Thrones.

Of course, if you’d rather listen to something shorter, here is a remix of Panama Wedding’s song All of the People, done by RAC.

Thank you so much for reading, and have a great weekend!

Language, Music, and Holidays

Sat, 19 Apr 2014 17:58:47 +0000

I am privileged enough to know a second language (although as the years pass, my proficiency is faltering…). The government and the military have a great need for foreign language proficiency for its employees (though apparently that isn’t much of a requirement for U.S. diplomats…). Given their need, they coordinated with the University of Maryland to develop a cognitive test that is supposed to determine how proficient someone can become in a foreign language. It may soon be publicly available, but honestly I don’t know if I’d be interested in taking it. While helpful as an aptitude test for job functions, oftentimes the interest and the attempt at proficiency is a great help for cultural relations with non-American countries. I’d be concerned that a test like this would cause people to give up languages earlier–if they know they’d never become fully proficient, why learn more than the basics or general education requirement?

In terms of making foreign languages more accessible, however, there is also the matter of translations. I’m currently writing about how language and national identity can have a tendency to segment the Internet, but it also has an impact on literature. One man wants to change that, by encouraging others to start their own publishing houses. He did, and focuses primarily on translated works from Russia and Central and Southern America, as he started his publishing house in Dallas, Texas. It’s a great read, with insights about the publishing business and notes about the commonality (or lack thereof) of translated literature in the United States.

And, well, if you don’t start a publishing house, you can always support your local bookstore. Janet Potter has a short essay about The Millions about her love affair with bookstores.

Pitchfork recently published a great longform essay on music streaming. It brought up a lot of great points, discussing the formats (radio, internet radio, spotify, soundcloud…) and the costs and tradeoffs of the various music streaming options. I have so many things to write about this, that it turned into a blog post. Beware that the pitchfork essay has adopted the all-too-common habit of feature web design, and made reading the article into a full screen experience complete with animated text. I wonder when (or if) the novelty will wear off for this magazine-style publishing.

In the same vein, these articles are somewhat older but call up similar themes. Bob Lefsetz calls out artists like Metallica (who supported RIAA in their copyright-infringement lawsuits), asserting that they need to decide which side they’re on as artists:

“We live in an attention economy, your biggest chore is getting people to listen, not to pay for your music. And the entire music industry is rotten to the core, riddled with egocentric businesspeople putting themselves first and responding not to music, but money.”

While services like Ponyo (which I’ve discussed before) are attempting to reclaim the sound lost by Mp3s and other lossy digital formats, audio engineers are cognizant of the need to master sound differently when many listeners will be listening in those low-quality formats:

“Mastering for iTunes was a different challenge,” VanDette told Ars. “You can’t get around it—when you throw away 80 percent of the data, the sound changes. It was my quest to make the AAC files sound as close to the CD as possible; I did not want them to be any more loud, hyped, or boomy sounding than the CD.”

The streaming essay talks a significant amount about how the trend toward algorithmically tailored music is also a trend toward more personalized, individual music experiences, potentially destroying the “imagined community” that can be developed through radio play (per Susan Douglas’ research). But headphones also had a significant impact on the personalization and individualization of music listening.

I’m cutting myself off otherwise I could talk about music all day. This week was rife with religious celebrations, with Passover earlier this week and Easter this weekend (and today is Good Friday). In honor of Easter, The Atlantic explores the atmosphere in Vatican City with the first retired pope, Pope Benedict XVI living only meters from the new pope, Pope Francis.

Additionally, The Rumpus has an essay about preparing the Haggadah, the prayers and thought-provoking readings that occur over Passover Seder. A holiday that promotes introspection and examination of where you’ve come from and what you’ve been through, surrounded by family and friends sounds like a really great holiday, even if the history surrounding Passover is a bit grim.

Anyway, for this week you can listen to this playlist of 5 songs by Daughter performed with an orchestra (or at least strings). It repeats itself for you so you can listen all day.

The Evolution of Music Listening

Fri, 18 Apr 2014 18:32:45 +0000

Pitchfork recently published a great longform essay on music streaming. It covered the past, history, and present of music streaming, and brought up a lot of great points. These are my reactions.

The piece discussed how “the “omnivore” is the new model for the music connoisseur, and one’s diversity of listening across the high/low spectrum is now seen as the social signal of refined taste.” It would be interesting to study how this omnivority splits across genres, age groups, and affinities. I find myself personally falling into omnivore status, as I am never able to properly define my music taste according to genre, and my musical affinities shift daily, weekly, monthly, with common themes.

Also discussed is the cost of music, whether it be licensing, royalties, or record label advances. Having to deal with the cost of music is a difficult matter. I wonder if I would have been such a voracious consumer of music if I hadn’t grown up with so many free options with the library, the radio, and later, music blogs. Now that I’m older, I make the effort to purchase music when I feel the artist deserves it, but as I distance myself (incidentally, really) from storing music on my computer, that effort becomes less important to expend.

Regardless of my own financial situation, the financial situation of the artist is important to discuss, and a Stereogum op-ed written by Kevin Devine of Of Montreal covers the politics that can surround that precarity. At times, selling out is viewed as something that no artist should do–compromising moral integrity, working for “the man”–something that is above the status of “musician” or “artist” that should be rejected. Fans, too, can come to dislike artists after they become too “mainstream”. But as Devine points out, that’s a false understanding:

“The idea that anyone who attempts to do anything commercial is a sell out is completely out of touch with reality.”

He continues, pointing out that this is even more ludicrous of an expectation in a capitalist economy:

“It’s impossible to be a sell out in a capitalist society. You’re only a winner or a loser. Either you’ve found a way to crack the code or you are struggling to do so. To sell out in capitalism is basically to be too accommodating, to not get what you think you deserve. In capitalism, you don’t get what you think you deserve though. You get what someone else thinks you deserve. So the trick is to make them think you are worth what you feel you deserve. You deserve a lot, but you’ll only get it when you figure out how to manipulate the system.”

And to get directly at the heart of the artists’ paradox, the expectation that they should not sell out, that they should provide their music or art to the public for free or cheap but also not license their music in commercials, is impossibly unrealistic:

“People who wanna be artists have the hardest time of it ’cause we are held up to these impossible standards. We’re expected to die penniless and insane so that the people we have moved and entertained over the years can keep us to themselves. So that they can feel a personal and untarnished connection with our art. The second we try to earn a living wage or, god forbid, promote our art in the mainstream, we are placed under the knives of the sanctimonious indie fascists. Unfortunately, there isn’t some grand umbrella grant that supports indie rockers financially and enables us to exist outside of the trappings of capitalism.”

Of course, there are grants from the NEA, but those are not large enough, plentiful enough, or valued enough by the government to be anything reliable for an indie artist.

DIY artist David Bazan discusses in the streaming essay that personal and untarnished connection with art as a main factor of why he doesn’t like digital music services:

“I’m like a small farmer who interacts with people who consume what I make and tend my little patch of ground," he says, “and the Spotifys of the world, which are like McDonald’s, are going to make people less aware of how the thing gets made and of its value.”

He values the fact that his fans will have that personal connection with his art, even if it means that he ends up penniless and relatively unknown. The loss of a conscious supply chain and the facelessness of the producers has long been something apparent in more manufacturing-based arenas, that has begun to go after more creative pursuits.

Is this really the case, though? Artists have long been producing art that is meant to live on its own. Though perhaps it is meant to live on its own with the understanding and celebration of who created it. Music may be somewhat different, because while in terms of classical music it is the composers who are remembered, but as of late it is the performers who are more often more memorable and tied to the musical experience. The Backstreet Boys were more famous than their producer. Digital music may only perpetuating this impression.

The ephemerality of digital music manifests itself in the listening experience even further, as the author discusses:

“Vinylphilia and other sorts of affective relationships with musical objects are nothing new, but they take on a new level of importance when contrasted with a transmission from a distant server rather than an mp3 on a hard drive. It’s often hard to understand the affordances of music media until the next one comes along. Mp3 files felt incredibly intangible, even ephemeral, compared to CDs, cassettes, and records, but at least you could carry around an object that you were certain contained those files. With the new crop of streaming platforms, there’s even less a sense of ownership, only the procedure of remotely licensing a digital file to start playing each time you click. It’s understandable that many artists worry about the implications of streaming music: Access isn’t nearly the same phenomenon as ownership. Internet radio platforms like Pandora highlight the long history of this idea.”

The interaction with music and the artists and performers shifts as the supply chain is masked and ownership of music is offloaded to the cloud and replaced by a service which offers access rather than objects. Not only that, but:

““The word “cloud” fills a similar function for SoundCloud, Apple, Amazon, and Microsoft as “platform” does for YouTube. It acts as a form of linguistic wallpaper masking the fact that digital copies of “your” music are in fact somewhere amidst vast server farms in places like Maiden, North Carolina; Ashburn, Virginia; Singapore; New South Wales, and Victoria, Australia; and Dublin, Ireland, subject to the terms of impossibly long end-user licensing agreements and to disappearance without notice.”

The supply chain is masked, but so is the storage of the music that you think of as “yours”–carefully curated spotify playlists, a vast collection of SoundCloud likes, or even 8tracks playlists crafted and uploaded to share. It can all disappear as quickly as MegaUpload was sued, thereby taking countless music blogs which hosted their music in that filesharing platform, offline.

“Mastering for iTunes was a different challenge,” VanDette told Ars. “You can’t get around it—when you throw away 80 percent of the data, the sound changes. It was my quest to make the AAC files sound as close to the CD as possible; I did not want them to be any more loud, hyped, or boomy sounding than the CD.”

The comments from “the public face” of Google Play Music, Tim Quirk, also resonated with me.

“Telling the entire world what it should and shouldn’t listen to has become far less important than simply making this overgrown musical jungle navigable,” he claimed. “Online music services need bushwhackers carving paths from one starting point to another. We’re not gatekeepers. We’re not tastemakers. We’re park rangers.”

I listen to most music on shuffle, and it is rare for me to listen to full albums. Having experimented with iTunes, YouTube, Pandora, SoundCloud, Spotify, and now Google Play Music, I treasure the shuffle and “instant mix” feature of Google Play Music, as it does a very good job of shuffling and creating something like iTunes’ new “Genius” mixes. This attitude is reminiscent of an idea brought up earlier in the piece:

“algorithmic-determined internet radio streams are not designed for those of us who already spend large portions of our free time (and work time) researching, acquiring, and sorting music. This has long been the divide between the “lean-back” approach to radio listening and the intense, deep-dive approach of aficionados.”

I have long been an aficionado, but now as I work full-time and my personal priorities have somewhat shifted, I don’t often have the time to invest in creating self-mixed DJ playlists or CDs for my car. I have fallen back to the “lean-back” approach of accepting what shuffle algorithms organize for me. At the same time, there is a risk in consuming some of these services, as:

“Chiding and rewarding these systems felt like a mix between topiary gardening and a Rorschach test: a non-stop process of pruning my tastes, spurred by my instant reaction to new information. Such acts of pruning, I realized, are perhaps the defining social activity of life in the Stream. Voting “thumbs down” on a track for not immediately satisfying me is akin to liking a status update or a tweet, or even voting on the likeability of friends themselves by “hiding” certain folks in my Facebook feed, or periodically treating my list of Twitter followees like an overgrown bonsai tree.”

To listen to some of the more active services that are attempting to recommend new music to you (not the greatest venues for someone like myself who grew up discovering music through college radio and music blogs) requires a time investment much like coping with Facebook’s new newsfeed algorithms. Algorithms have to be trained, and that training process can often be just as much a cognitive burden as deciding what sorts of music to play yourself. Services which offload that algorithm training through an analysis of your music library (iTunes, Google Play Music) will likely find more success in the future.

Of course, in my own experiences managing a library much larger than I could possibly listen to, I made ample use of the ratings to craft my own curated playlists and listening experiences out of the overwhelming amount of music on my hard drive. Thankfully, that was far smaller than the music available online today. While listening to music is seems socially integrated now, and fandoms hold strong as any concert attendee has witnessed, historically, radio was a powerful connector. As Susan Douglas, a radio historian describes,

“broadcast radio added “a new cognitive dimension” to social life that could bind “utterly diverse and unknown people together as an audience,” forging “powerful connections between people’s inner, thinking selves and other selves.” What is often called a “monoculture” nowadays is described by Douglas as an “imagined community” of strangers, united not through shared space, but because they were hearing the same programming (and ads) at the same time. Internet radio breaks sharply from this model, building itself around the individual, not the mass, and scraping activity data into a hyper-personalized experience.”

More recently the trend toward algorithmically tailored music is also a trend toward more personalized, individual music experiences. Perhaps this is somewhat due to the impact that headphones have had on the personalization and individualization of music listening, but digital music recommendation services have furthered this into the software and music marketing industry, rather than the hardware industry of headphones and walkmen that existed and evolved in the past few decades. The algorithms and “music genomes” that define the music we listen to today are at risk of eliminating the “imagined communities” that Douglas associates with radio listening:

“Do taste topiaries run the risk of sealing listeners into self-flattering cocoons? Is the tendency toward diversity—determined on internet radio stations by a knob that slides between familiarity and “adventurousness”— mitigated by the safety of algorithmically-determined similarity? Is this something to fear, or just the big data version of niche marketing tactics that took hold on radio decades earlier? Does internet radio create a virtually cloistered local scene for the age of infinite “curation”? A Top 40 Möbius strip to each individual?”

However, the essay doesn’t follow this much further. Spotify and SoundCloud both have social integrations into their experiences, with Spotify having a heavy Facebook-infused social sharing and following element. Soundcloud has been fleshing out their social venues, but allows interaction across a common stream of music posted or reposted by those you follow. There is a real risk, however, that these digital music services are creating a filter bubble in the music industry. It’s less common that you could pull up next to someone at a stoplight and both be listening to the same song in your car. However, maybe that just means that we’re both listening to music that we really, truly enjoy, instead of what is being filtered through the radio. The music industry has changed so much over the past century that I’m not going to attempt to predict what will happen or what could come next.

Whatever happens, I’ll be listening.

Heartbleed, Borders, and Cookies

Fri, 18 Apr 2014 17:03:30 +0000

HEARTBLEED heartbleed my heart is bleeding about heartbleed….

How soon until someone writes a country ballad about heartbleed? Knowing the Internet, probably before all the [currently vulnerable sites](broken link removed) are patched. Researchers at University of Michigan previously produced a tool which was capable of scanning large swaths of the Internet at incredibly fast speeds. They took advantage of this tool to regularly scan the top 1 million sites on the Internet ([as categorized by Alexa](broken link removed))(who is not a person) and determine what portion of the sites are vulnerable. Mashable, meanwhile, has compiled a [list of the big websites that were vulnerable](broken link removed) (but now are not). This bug is the latest and greatest of them….yet (as XKCD points out)

As the NYT points out, as the web gets larger it also gets less secure (and thus, harder to defend):

“If you fix one Internet security bug, you can be sure that attackers will just find another, potentially more dangerous one. “Over all, attackers have the competitive advantage,” said Jen Weedon, who works on the threat intelligence team at the security company Mandiant. “Defenders need to defend everything. All attackers need to find is one vulnerability.””

Rusty Foster, writing for the New Yorker, eloquently makes user-friendly sense of the bug (something I admire, working as a technical writer) while also taking care to call attention to the risks of relying on open source projects in their current state, as well as aging digital infrastructure:

“Unlike a rusting highway bridge, digital infrastructure does not betray the effects of age. And, unlike roads and bridges, large portions of the software infrastructure of the Internet are built and maintained by volunteers, who get little reward when their code works well but are blamed, and sometimes savagely derided, when it fails. To some degree, this is beginning to change: venture-capital firms have made substantial investments in code-infrastructure projects, like GitHub and the Node Package Manager. But money and support still tend to flow to the newest and sexiest projects, while boring but essential elements like OpenSSL limp along as volunteer efforts. It’s easy to take open-source software for granted, and to forget that the Internet we use every day depends in part on the freely donated work of thousands of programmers. If open-source software is at the heart of the Internet, then we might need to examine it from time to time to make sure it’s not bleeding.”

Old programming languages were more intensive, and more hands-on, and thus requires that you remember more things. In this case, the software was written in C, which requires that the programmer remember to manage system memory with the code. This high-level amount of management is possible to keep track of, but given that OpenSSL is wholly volunteer driven (as Foster points out), it’s much more difficult when that work is your hobby. The Sydney Morning Herald talked to the programmer who wrote the vulnerable code.

And, for the last of the bleeding hearts, Charlie Warzel writing for Buzzfeed takes care to remind us that the biggest issue with this vulnerability (besides the fact that it’s been unpatched for more than two years) is the future risk if sites aren’t patched. As an example:

““If one guy is running a soccer blog for his kid’s soccer team and doesn’t patch the bug, some attacker can come in down the line and comprise the site and put a virus on that will attack visitors,” Wisniewski said. “The big sites are almost all fixed or will be soon. The real concern is for the future.””

If you’re totally at a loss with how you could possibly come up with some new unique passwords, The Hairpin has your back (and so does the free, trustworthy password manager LastPass)

The Internet is often spoken of as something that will reduce the reliance on borders. However, in the physical world, borders matter more than ever, at least for those who cross them.

NPR has developed a beautiful web app that walks a person through various experiences on the United States - Mexico border. Don’t click because it’s beautiful. Click because it’s important, and tells stories that need to be heard more often.

New media site [Vox also delved into border issues](broken link removed), with a short look at the use of (unnecessary) force by border agents.

That’s enough of depressing things, though. While I’m not a great cook, I love baking. I only managed to make chocolate chip cookies twice this past year, but they are possibly my favorite cookie, and you can read all about the history in the New Yorker. But if history isn’t your style and you’d prefer a deep and scientific understanding of what gives chocolate chip cookies their characteristics (and how to adjust them to be more in line with what you want), you should read this in-depth post on the science of chocolate chip cookies. And if you don’t care about the history or the science and just want to bake, well, the recipe is here.

Spring is finally here, and brought with it the perfect weather for driving with your windows down. In the spirit of that power, some upbeat music by George Barnett with 3 Statues.

Work, Sleep, and Accessibility

Fri, 04 Apr 2014 20:00:42 +0000

Here’s what was important this week…

I went home sick yesterday. Even though it is a good decision for my health, I still felt bad leaving work. Often I feel like I might be more productive though, working different hours, or even less hours. Other countries allow for leisure time throughout the work day, like a two hour long extended lunch. America, despite the increasing efficiencies produced by a continuing offloading of human work to machines (computers, robots, mechanization), seems destined (doomed?) to continue down the habitual path of an 8-hour work day (with potential for more work or availability, depending on the profession).

This article from 2010 points out that Your Lifestyle Has Already Been Designed—this isn’t likely to change:

“the 8-hour workday is too profitable for big business, not because of the amount of work people get done in eight hours (the average office worker gets less than three hours of actual work done in 8 hours) but because it makes for such a purchase-happy public. Keeping free time scarce means people pay a lot more for convenience, gratification, and any other relief they can buy. It keeps them watching television, and its commercials. It keeps them unambitious outside of work.”

As the New Statesman has pointed out more recently, workers should exercise the right to be lazy, as the Cult of Hard Work is Counter-Productive.

“We are everywhere enjoined to work harder, faster and for longer – not only in our jobs but also in our leisure time. The rationale for this frantic grind is one of the great unquestioned virtues of our age: “productivity”.”

However, if you take a little time out from your work to check Twitter, zone out, or play a game online, you don’t have to feel that guilty, because “People who engage in “workplace Internet leisure browsing” are about 9 percent more productive than those who don’t.” There is, in fact, a way to waste time properly.

Not only can this pervasivity of a hard-working ethic be harmful, but the New Statesman asserts that “Our world has become an ambient factory from which there is no visible exit and there exists an industry of self-help technologies devoted to teaching us how to be happy workers.“

At the very least, our society continually values productivity over sleep. This is especially notable in “startup culture” in Silicon Valley, which practically fetishizes the incessant dedication of an employee that works while they sleep.

Indeed, for that industry, “Sleep only postpones the future and all the productivity it could hold.” But as this Forbes essay continues to argue, this sleep deprivation is producing a high failure rate of startups:

“Could it be that the myth of the obsessive careerist whose dedication to work follows him to bed every night is actually a grand farce of worst practices and general dysfunction? It may be that accepting the normalcy of non-stop work is encouraging a culture of unusually bad thinking, painstakingly propped up by those charged with turning thought into real product.”

The rediscovery of the importance of sleep, one could call it.

This has been studied a bit more increasingly, in a new field called chronobiology. I first learned about it by reading Douglas Rushkoff’s book Present Shock, and this review in the Rumpus details the concept as he illustrates it:

“Chronobiology, a buzz word term for a science not yet fully developed, studies the impact of trying to live our circadian cycles in a 24/7 world. Rushkoff uses the very real syndrome of jet lag as an example of the physiological effects of digital time exceeding our biological parameters. No matter how fast the machines around us move, we’re still built for a world that turns on seasons and the movement of prey — at least for now.”

One town in Germany, Bad Kissingen, is going so far as to actually design their town around sleep schedules, to the point of collecting the townspeoples’ “chronotypes” and use that information to reformulate school hours, among other systems.

Having spent a few years in the workforce now, I’ve realized that I work better when I get to sleep in later and stay up later, but standard hours are standard hours. Having a set schedule, does help, but I can’t help but wonder how my life would be different if, say, more restaurants were open past 9 or 10 to accommodate people who go to bed later (probably wouldn’t happen unless more people were able to sleep in later…)

Museums are a cultural institution. A destination when on vacation. A great way to spend a Sunday afternoon. Inaccessible to many with limited abilities. But that has been changing. A Dallas art museum has been making efforts to be more inclusive to the low-vision community. But “The museum doesn’t want its outreach to the low-vision community to be in the form of exclusive activities or hours; instead, it hopes to attract a range of learning styles.” And it’s having the desired effect. As museum’s education director, Carmen Smith, describes, “A lot of our national inclination when we describe a work of art is to leap right into what the work is about and what it means instead of telling someone what it looks like and letting them make their own interpretation.”

The Metropolitan Museum of Art in NYC is also joining forces with Parsons the New School for Art and Design to discover new ways to improve accessibility in museums. Just like the Dallas program, many of the results are beneficial to people beyond the disabled, such as an app that lets you create your own route through a museum based on the art that you want to see. This allows people with low-mobility to conserve their energy while still seeing all the art they want to see, but would also allow busy people to make more use of a tight museum schedule while traveling. (Of course, wandering is not to be discounted in value!)

Even as museums make more of an effort to be accessible, there is still work to be done in presentation of some art. As recounted in a blog post by museum two:

“I remember a photography exhibition in Boston where one photograph of three young ballerinas was labeled with their names. A second image, of three ballerinas with Down Syndrome, were labeled with their difference. The message, when museums produce targeted campaigns or events or exhibitions for non-white audiences is: we acknowledge you as others in our midst. Not as humans, or artists, or scientists, or dancers. As others."

Libraries also make efforts to improve accessibility, (and they are also cultural institutions devoted to learning, much like museums). As this tumblr post describes in detail, little things like an automated hold system that places all holds in a central pickup location allow for “one-stop-librarying” for not just low-mobility patrons, but anyone with a library card. Again, another benefit that makes a huge difference for people with low-mobility that is beneficial to everyone that uses the library.

One designer is making significant effort to bring the inaccessibility of places to the attention of able-bodied people. One of her projects, a portable ramp, is great for use in “older cities, like Boston and New York. Many storefronts have a single step separating their entrance from street level, and though the ADA doesn’t require any accommodation, that step can be a significant obstacle to anyone in a wheelchair. Or, for that matter, anyone with a stroller, or a walker, or crutches, or a heavy suitcase, or a cart with a delivery to make.” It’s a wonder that inaccessible places are so common, because the changes and efforts made to improve accessibility not only allow a disenfranchised population to gain access to so many great resources, but the everyday life of everyone becomes a little bit simpler.

This week’s music recommendation is Zella Day with her song 1965.

Thanks for reading!

Noise, Medicine, and Music

Tue, 01 Apr 2014 17:41:10 +0000

Here’s what was important this week…

More than you probably ever wanted to know about refrigerators and refrigeration:

“Refrigeration is the invisible backbone on which the world’s food supply depends — and given our climate-changed forecast of more extreme weather events, it may yet prove to be its Achilles’ heel.”

Oh how I wish this had come true:

“All mechanical fridges work by controlling the vaporisation and condensation of a liquid called a refrigerant. Most fridges today do this control with a special electric-power pump called a compressor, but there’s also the technique of absorption, which is kicked off by a gas-fuelled flame. The fridge’s hum wasn’t inevitable.”

I have somewhat of an aversion to background humming noises, like that of a refrigerator, central air system, fluorescent lights, or washing machines.

Constant noise like that is only more noticeable when it’s gone. I used to work at an IT help desk, and the hum and noise of that environment can range from a steady hum, to a cacophony, to the single solo jarring voice cascading across the cubicles. This essay by Nicole Dieker about her first job as a telemarketer aligned somewhat closely with my experiences, although gladly my job was a bit more welcoming and less restrictive than most call center environments.

Many call centers are outsourced, however, and that trend is continuing to grow. As it does, it is becoming more nuanced but expansive at the same time. While a cost-cutting measure, it concerns me if companies outsource all of their IT resources, then neglect to align business process improvements with IT upgrades and instead languish with the same technology for years, as it becomes more insecure, outdated, and in the case of the Windows XP operating system next week, unsupported.

Electrocuting instant film

Medical care is something that is in the news often, with the deadline for enrolling on Healthcare.gov looming. Improving the capabilities of human-powered medicine is paramount, even as we move closer to an augmented reality full of robot-human interactions (already rife with computer/algorithm-human interactions). Something as simple as implementing consistent checklists help to improve patient care and hospital efficiency, and now more attention is being paid to altering hospital culture to further these same goals. As Ilana Yurkiewicz’s essay in Aeon points out:

“Brutality doesn’t make better doctors; it just makes crankier doctors. And shame doesn’t foster improvement; it fosters more mistakes and more near-misses. We know now that clinicians working in a culture of blame and punishment report their errors less often, pointing to fear of repercussion. Meanwhile, when blame is abolished, reporting of all types of errors increases.”

Culture isn’t just important within a hospital staff, it is also important when caring for patients as well. Leslie Jamison recounts her work as a medical actor, helping to evaluate future doctors on their patient care, intertwined with a narrative of her personal experiences in the hospital:

“Empathy is a kind of care but it’s not the only kind of care, and it’s not always enough. I want to think that’s what Dr. G was thinking. I needed to look at him and see the opposite of my fear, not its echo.”

From a patient’s perspective, Will Boast’s father stoically dealt with pain his full life until it killed him:

“During a long ordeal or a long depression, we begin to feel pain more acutely; we only learn to show it less. The stoic’s creed, the stoic’s prayer—what doesn’t kill you makes you stronger; bear down and take it, you’ll get through; keep calm and carry on—it all turns out to be nonsense.“

Geoff Dyer survived a stroke, which struck him young despite a healthy lifestyle, and recounts his experiences in the hospital:

“I changed into one of those hospital gowns that tie at the back, the purpose of which seems to be to enfeeble you, to reduce your capacity for independent action. To walk even a few steps is to risk the ignominy of exposing your bottom to the world. You are now a patient, the gown decrees, the recipient of treatment, someone to and for whom things are done.”

Last week I was enamored by Billy Joel and Jimmy Fallon’s rendition of The Lion Sleeps Tonight, and I later discovered the full story behind the song. While there are lots of rumors about what the song lyrics reference, according to essayist Rian Malan, the lyrics were simply fabricated by an American man in order to transform the melody of the original, South African Zulu song, into something more mainstream and accessible for the music charts. Malan’s essay has a lot of great information about the original song, the version performed by Pete Seeger and the Weavers, and the mainstream popular song performed by the Tokens. Not only does he manage to cover the full history of the song, he also captures a history of whitewashing African and African-American music.

On the note of whitewashing history, history can also be “man-washed”, if you want to call it that. The history of beer-brewing is one of many industries which were formerly considered the lowly domain of women, which later (with the industrial revolution, return of men from wars, or popularity of the field) become dominated and overtaken by men. A Medium collection looks at experiences of being “The Only Woman in the Room”, and this essay about women brewers is included.

I witnessed performances by Fred Thomas, formerly (currently?) of Saturday Looks Good to Me, and Mister Suit last night. Fred Thomas reminded myself and my friends collectively of Atom and His Package, alongside Andrew Bird. Mister Suit had more of a British aesthetic while performing, reminiscent of The Hives or Franz Ferdinand, but sounded much like older Kings of Leon, like songs on their album Because of the Times. Both artists performed alone, with just a guitar and their pedals.

Physics, Social Media, and Data Privacy

Sat, 22 Mar 2014 17:19:25 +0000

Here’s what was important this week…

A discovery was made at the South Pole BICEP2 research lab that, if accurate, confirms the theory of cosmic inflation. Cosmic inflation explains the big bang, as an article in Symmetry Magazine details: “Almost 14 billion years ago, the universe we inhabit burst into existence in an extraordinary event that initiated the big bang. In the first fleeting fraction of a second, the universe expanded exponentially, stretching far beyond the view of today’s best telescopes. All this, of course, has just been theory.”Cosmic inflation and the big bang are neatly explained in more detail in this PHD comic.

My favorite part of the discovery is a video released by Stanford of one of the scientists in the BICEP2 project going to visit one of the lead theorists of cosmic inflation to tell him that his 30 years of work had just been confirmed. Much more footage was taken, but they edited the video for emotional impact--and it worked (I teared up). Such a video turned out to be an easy way to make the discovery go much more viral and affect many more people than it otherwise might have. Wired details how the scientists involved manage to keep their discovery quiet until the announcement.

For another story from the field of physics, “organizers of a biweekly forum on Large Hadron Collider physics at Fermilab banned PowerPoint presentations in favor of old-fashioned, chalkboard-style talks.” They found that “Without slides, the participants go further off-script, with more interaction and curiosity,” and doing so also allowed more involvement of the audience. I think this is a great idea, as powerpoint and “slide decks” tend to dominate presentations across many industries (certainly including mine). While slide decks (see examples on slideshare) can be helpful to convey the information in a talk to an audience that was unable to attend, an actual recording of the talk (with the questions that the audience asked included) would always be more valuable. While that requires resources that many often don’t have, I’d think that annotated, but content-less slides (much like this presentation transcript by Frank Chimero) would be far more beneficial. Those still require a substantial time investment, but allow for context-specific content to rule: a talk with a whiteboard or slides of demonstrative images, and a webpage or paper transcript that incorporates images with the text of your speech. Rebecca Schuman goes into more detail (in Powerpoint format!) about why Powerpoints are bad for teaching and presentations.

Now that it’s Spring, that means one thing! Since it isn’t actually warm outside, you can instead take 5 minutes to clean out the apps that you’ve authorized to access your various social media accounts! I tend to be vigilant about this, and set it up as well (I no longer remember how) so that even if an app on Facebook somehow gets granted the ability to post on my behalf, those posts would be visible by only me. These app authorizations are an important capability, however, and one I appreciate. The use of social ids to log into a service cuts down on the number of passwords we have to use (and thus reducing chances for password re-use) while allowing us simple and easy access to more services. Of course, that does mean as well that if your password is compromised, access to more than one service has been interrupted. That’s one reason why you should always choose a secure password, and also why social logins tend to be used on low-stakes products like a book review service, or another social media site, and others.

The services benefit as well by improving customer engagement (more sign-ups) and by taking some of your data from the social account that you used to log in with, but lose an element of authority over your relationship with their service or product.Yahoo, for example, will no longer allow people to log in to its services with social accounts--they must create a Yahoo account to continue using them. Yahoo is framing it as something which “will allow us to offer the best personalized experience to everyone", which really means that they want the ability to store identity information and data about you to serve you personalized ads, just like Google and Facebook do.

In terms of improving privacy of people using its services, Google has just announced that all connections related to Gmail will be encrypted, or protected on the network. While it was already something that you could enable, now it will be by default, and be encryption across the entire data transmission. This means that any data sent from Gmail, whether it is an email, a photo, or the data that Google picks up from your browsing habits within your email, will be protected as it travels across the network from your computer, to the servers in Google’s datacenters, between those datacenters, and on to the computer of the person you sent the message to (assuming they also have Gmail, in this example).

Another way to increase security on the web is to treat it like a public health problem.

“People who worry about security and freedom on the internet have long struggled with the problem of communicating the urgent stakes to the wider public. We speak in jargon that’s a jumble of mixed metaphors – viruses, malware, trojans, zero days, exploits, vulnerabilities, RATs – that are the striated fossil remains of successive efforts to come to grips with the issue. When we do manage to make people alarmed about the stakes, we have very little comfort to offer them, because Internet security isn’t something individuals can solve.”

This issue is similar to the challenges of public health officials trying to convince people of the utility of vaccines. As the Guardian article continues, “when it comes to public health, individual action only gets you so far. It doesn’t matter how good your water is, if your neighbour’s water gives him cholera, there’s a good chance you’ll get cholera, too. And even if you stay healthy, you’re not going to have a very good time of it when everyone else in your country is [stricken] and has taken to their beds.” A concerted, government-sponsored effort is often the only way to improve the health of the citizens and the environment, and if improvements are undermined by the government or other powerful groups (as the Taliban in Pakistan and Afghanistan demonstrate with polio vaccination efforts, and the NSA/GCHQ demonstrate with IT security efforts), then individuals will remain at risk.

Another way of improving security would be to punish hackers differently. This article is from Australia so the laws it references are different from American laws, but the same principle applies. The United States currently focuses punishment on the tools and the methods that were used to commit a crime, with the Computer Fraud and Abuse Act (CFAA). One thing that the law criminalizes is an act which “intentionally causes damage without authorization, to a protected computer” However, as the article recommends, it would be far better to think about “how criminals use technology to increase the harm they do, just as we already do when we punish acts of violence.” Thus, you’ve committed a crime of assault if you beat someone up, but aggravated assault if you do it with a weapon. This practice “shifts the focus onto the impact a crime has on the victim and the impact of using technology rather than the way that technology works. That means the legal system does not necessarily have to understand the technical details of a technology in order to reach a decision, it just has to look at the effect it has had.” (emphasis mine). If you’d like more information about CFAA and its functions as a law, watch this presentation by Jonathan Mayer on “Terms of Abuse”. (his bio is here)

Jimmy Fallon is the new host of the Tonight Show, and he’s been doing fairly well. While I don’t watch cable and thus haven’t seen his show (and would probably be too sucked into an NCIS or House Hunters marathon to bother), but it seems to be a success so far. I’d say his success is likely due to his inclusion of music in the show. Not only does he have The Roots as his guest band, he also collaborates with his guests to do original renditions of hit songs (Idina Menzel with Let it Go) or to fulfill a dream of his and sing the Lion Sleeps Tonight with Billy Joel. (Billy Joel is also pretty great at fulfilling requests like this, like when he let a student accompany him at a Q&A session). By cutting these clips into youtube-digestible and easily shareable YouTube videos, he amplifies his success beyond cable to the Internet, letting the Tonight Show compete anew with The Daily Show with Jon Stewart and the Colbert Report.

As always, thanks for reading!

My Photo Here

Thu, 20 Mar 2014 16:22:06 +0000

In the spirit of @weboesel and being true to my whole identity, I am adding a picture to my blog and my twitter account.

I’ve had a neutral (non-person) picture on both services since I joined. I relish the implicit neutrality that this sort of picture offers me, but I’m choosing to assert the whole of my identity across the services I inhabit.

A neutral picture lends an element of gender-neutrality to the twitter timeline, as there is no face with which to implicitly associate with stereotypes. I feel that without an identifying photo, I am more able to remove associations of my gender from interpretations of my work, although my name is prominent on these services. Perhaps I am merely being overly guarded against potential sexism. There are many other reasons for not using a personal photo online, but this was mine for these sites.

I made the decision to use my own photo, however, because I want to own my full identity. I don’t want to hide my woman-ness (as a proud feminist) for fear of not-yet-existent sexism on these services. That is just another form of self-censorship, in my case. In addition to owning my identity (and letting my true self “shine” in the words of a friend), a photo allows me to more capably link my public identity across services. I will maintain pseudonymity through usernames on other services, but on the services which I identify myself with my full name, I will include a photo. This also allows, in the spirit of Keybase.io, a further layer of identity vetting and verification.

Here’s hoping for the best.

What is Old is New Again: Following Up

Sun, 16 Mar 2014 20:15:59 +0000

Books are holding their own against e-books, vinyl is making a comeback, film if it isn’t making a comeback, is holding steady since the Kodak bankruptcy. This cycle gives voice to what we want from our technology, devices, and everyday contraptions. In my first post on this matter I reasoned that it comes down to the experience, and that “The contrast between a digital and an analog experience can alter interpretations of media.”

This still holds true, and the nostalgia and authenticity attached to an analog experience has led digital technology to be reworked, in a way, to take advantage of these emotions.

Books

Books and e-books can work in tandem, but the medium might matter more when it comes to comprehension: "“We bombarded poor psychology students with economics that they didn’t know,” she says. Two differences emerged. First, more repetition was required with computer reading to impart the same information. Second, the book readers seemed to digest the material more fully.” It seems fairly true that “it may be harder to grasp material consumed in e-book form, where the words slide by as if on ice skates, than in print.” The tangibility of paper books, then, has real and established value for memory and learning.

The role of design in books has been hurt by e-books and the role of technology, but perhaps that has more to do with platform design: “The more we are able to push and pull content in and out of the cloud, the more it seems that the content itself lacks an inherent visual identity. In tangible terms, that means that a book, though painstakingly designed inside and out, may carry none of its visual properties with it when it’s rendered on a Kindle.” A book is painstakingly designed as a book, but perhaps if it were painstakingly designed as an e-book, the opposite challenge would emerge. Can an e-book be designed in such a way that its design can be translated in print as well?

The same article continues, “The more that digital becomes the native format of content, the more its influence will be apparent on analog content”. According to this essay, paper books will work harder to emulate digital e-books as those become the norm. I think it’s much more likely, however, for paper books to embrace their format and the value and design that paper allows, such as a book cover for Fahrenheit 451 where the spine is made of matchbook striking paper. This sort of design cannot be replicated in digital form, as it requires a certain tangibility to be meaningful. Digital books as well, would need to embrace their format as a virtual storytelling medium that allows techniques such as hyperlinks, and some form of engagement. E-books can be used to end a passive engagement with a story, as:

“Books are on the rich but passive end of the spectrum. Multiplayer games are on the other end of being interactive but haphazard. How can we make use of technology to deliberately design a truly interactive system that connects many people’s consciousness in a plot that flows like a book with some sense of a story?”

That can certainly be done with an e-book: a choose-your-own adventure book would no longer be limited by the number of pages.

An additional influence of e-book design on book design could be to transform the role that books play in our lives as objects. Indeed as this Slate essay points out, “the book has always been as much a design object as a vehicle to enable the experience of reading. (In fact, publishers are fighting the physical book’s demise with a renewed effort to use innovative methods to make books into beautiful objects that we want to touch and hold.)” Paper books can take advantage of their tangibility status and become closer to works-of-art to be displayed, while e-books lend themselves to the more private business of pure consumption.

The function and format of books have always influenced and driven their design. A book from 1759 is nearly the same size as a book displayed on a smartphone screen. As Clive Thompson recounts, “That small-page format was quite common back in the 18th century. It’s known as octavo duodecimo — with pages that are about 6 inches by 9 inches. The entire Conjectures is only about 8,000 words long, but it was common to print essays in this pretty little style, because it had great ergonomics: It made for easy one-handed reading and portability.” As he examines why that format grew out of style, he points out that with the rise of hardcovers, there was the need to produce weighty books. However:

“If you don’t need to charge $25 for a book, you don’t need it to necessarily balloon to 300 pages, so you can begin to publish books that look more like … Conjectures or The Art of Memory. The book fits the length of the utterance, instead of expanding to fill a vessel the dimensions of which were determined by haptic marketing: “Yeah, this book feels weighty. I’m in!””

Thus, “the ergonomics of smartphones as reading devices are not only kind of rad, but historically so.” This historical fluctuation of format based on consumption and function means that for the same to happen lately isn’t too far of a stretch. Indeed, small, pocket sized books exist very cheaply in Germany (can’t vouch for elsewhere), but less often in America.

Some objects have inherent value when rendered in analog format, and should continue to flourish regardless of the influence of the digital. A new project called Wink Books by some of the folks behind Cool Tools works to celebrate books that “deserve to be printed on paper”. As the people behind Wink Books put it:

“Wink scours bookstores, libraries, flea markets, and online retailers looking for books that you must experience; books that are sensual, three dimensional, robust. We seek out artifacts that you must hold in your hands or unfold in your lap. Wink collects books that optimize what books do best on paper: open up new worlds. Our test for Winkdom is simple: would this book work as an ebook? If yes, we ignore it. Rather we gather for you the best books that work on paper. Wink books will never go obsolete – they can still be enjoyed in a hundred years. As much as possible, we go out of our way to find paper books that are little known treasures, that are uncommon and unconventional, and yet are still available (that is, we don’t feature “rare” books).”

The allure of (analog) books over electronic, digital books is in the experience: how these books appeal to the senses in ways that digital books are incapable of reproducing.

Many new media formats are experimenting with the old format of serialization and short-format to bring some of the interaction and the “newness” back to books and storytelling. Twitter is a popular medium for storytelling, surprisingly, “because of the intimacy of reaching people through their phones, and because of the odd poetry that can happen in a hundred and forty characters.” Teju Cole has been embracing the limitations and techniques in this format, most recently with his story “A Piece of the Wall” crafted in a custom timeline.

Beyond Twitter, apps such as Rooster employ serialization as well: “The app’s approach is to deliver novels in bite-sized chunks. Each segment should take about 15 minutes to read, and they arrive when you choose, whether it’s every morning before work; every Monday, Wednesday, and Friday; or whatever. There are two books offered every month — a contemporary title and a classic novel that’s supposed to offer “a conversational counterpoint.””

Rooster isn’t the only app out there, as ““Amazon announced a new Serials program” last fall, in which novels are published an episode at a time, rather than all at once. A startup called Plympton ran a successful Kickstarter to support its own serialized fiction efforts (it’s also partnering with Amazon). And serialization is part of the model at another digital publishing startup called Coliloquy.” This trend operates counter to the “broader conversation around how media consumption is changing, with no one willing to wait for anything anymore. At least, that’s what I keep hearing and reading: Everything will be released everywhere immediately because waiting is so 20th century, man.” Serialized fiction and these digital techniques recapture the Victorian patterns of Serial Literature and use it to remake the stereotypes surrounding digital media and storytelling that have been thrust upon them by Netflix-esque binge-watching.

As serialized books were once popular and return today, so too does the semi-private newsletter. “The last year or so has seen an intriguing renewal of a genre from the early years of the internet: the email newsletter. A couple of months ago Alexis Madrigal described this development as a natural and healthy response to a never-ending and increasingly vast stream of online data”. While the past involved similar content, as “In 1753 Grimm began to write and edit La Correspondance littéraire, philosophique et critique, a newsletter about cultural events that was meant to supplement and when necessary correct official news sources, which were subject to rigorous censorship.” the purpose of writing newsletters has changed dramatically in some locales. Once used to subvert censorship and provide an alternate source of news, newsletters now serve as careful curative digests of the news, seeking to find the importance (or the intriguing, in the case of Madrigal), of the firehose of the web.

Although books may help you retain content better than e-books, some countries are digitizing all of their books, but in the United States, the paper lobby is discouraging the ever-increasing move to digital. This may seem backward or counter to progress (indeed, the federal government is one of the larger users of some of these old technologies), but in fact too little attention is being paid to the environmental costs of producing electronic devices and disposing of them, such as batteries and rare earth minerals, not to mention the lack of universal and affordable access to the Internet(link covers only United States). The environmental impact of paper, at least, is far more consumer-transparent.

Nostalgia surrounding technologies replaced or subsumed by the Internet and the web is encapsulated in the recent Paper advertisement for Facebook’s app:

“It’s curated, just so, to replace the look and touch of technology as we know it with symbols from the past. The letter for the email, the Polaroid for the JPEG, the typewriter for the laptop. Each of these vignettes portrays a creative act we know, yet are far more likely to do digitally, happening in a world we know, yet is far more likely to be filled with synthetics and throwaways.”

Yet as Chris Butler continues, he takes to task what is hidden in advertisements and nostalgic-navel-gazing practices such as this: They “don’t show you the massive, energy sucking data centers behind the tech in their future worlds — the 40,000 megawatt blocks of nothing but snarls of cable and plastic. But they’re there somewhere. They have to be, unless they figured out some other way to store all our selfies and status messages, and some other way to keep the lights on besides burning things.” These environmental impacts of our digital tech will become, and must become a larger part of the conversation surrounding digital technology.

Art

Paper (the medium) “is a star of this Biennial, with dozens of books and printed material. “Now that we have access to more archival material, we are all preoccupied with how we can reanimate it and create living histories,” Mr. Comer said.” The Whitney Biennial, which attempts to portray a slice of contemporary art movements, is in part celebrating the function of paper. “The first Arts and Crafts movement, in England, challenged the taste of the Victorian era. Now the handmade aesthetic is flourishing again, Ms. Grabner said. “As so much moves to the digital world, there is a movement of slowing art and life down.”” The trend has been to recapture the old, to slow down our connections and our consumptions, and understand it anew through art and handmade pieces.

Another paper element of the Biennial is the leporellos, which

“In his catalog essay, Comer calls the unfolding pages of the leporellos “a proto-screen, a kind of precursor to the laptops, smartphones, and tablets that increasingly dominate our lives, where the distinction between language and image continues to collapse and multiple surfaces and screens abut and fold into one another.””

Why has paper (and words) been so celebrated as an art form in this Biennial?

” I have a theory why this is so. As the practice of writing on paper (everything from telegrams to letters to books to Post-It notes) is increasingly devoured by technology, words on paper are evolving from widespread tools of communication into the rarefied stuff of art. As things recede, they also expand. As a result, words are becoming as legitimate as the more traditional subject matter of painting, drawing, video and sculpture. Running parallel to this trend is a more capacious notion of what constitutes art.”

This role of paper, handmade art, and words as “older” technologies lends themselves as valuable works to be enshrined in museum exhibits.

As some museums treasure the role of paper, handmade art, and words in contemporary art, another (online) museum is making painstaking effort to preserve “vanishing and endangered sounds” such as “The sound of a dial telephone, a walkman, a analog typewriter, a pay phone, a 56k modem, a nuclear power plant or even a cell phone keypad are partially already gone or are about to disappear from our daily life.” The online museum also makes the effort to accompany the sounds with interviews, so as to “give an insight in to the world of disappearing sounds.” This museum functions as a repository for the old, the vanished, and the nostalgia for outdated technologies.

Music

Beyond miscellaneous everyday sounds, music has become a medium in which artists and listeners alike make the effort to recapture what has been lost in the digital era. Vinyl is experiencing a resurgence, as I’ve discussed before, to such a degree that Pioneer teased a new turntable model at Musikmesse 2014. There is even a vinyl-by-mail subscription service if you’re inclined to partake.

Vinyl has certain medium-specific quirks that artists can take advantage of. While in digital music-making, vocal pitches can be tweaked by software, and some DJs use laptops instead of record players, vinyl as a medium allows techniques like scratching that can be falsely replicated by digital sounds. It also lends itself well to other playful functions, like an element observed in this passage by a woman who listens to all of her husbands vinyl records and blogs her impressions:

“The last song, “It’s Obvious” repeats the phrase “you’re equal but different” over and over again at the end. WOAH then it sounds like it’s over but it starts again, but then just keeps playing over and over again like it’s skipping! But it sounds intentional! I just said to Alex, “is this skipping or is it supposed to sound like this?” He asked me if the needle is on the last groove of the record, which it is. Then he said, “There’s a thing called a lock groove, sometimes people put a repeating phrase on the last groove and let it play over and over again.” Me: “That’s awesome!!” He also told me that some times artists will etch secret message on the “inside ring” of the record, but we looked and this one just says “Birmingham” and Belfast, Bristol, Brixton.” If that is a secret message, I don’t know what it means. Still pretty cool though. I’m really into lock grooves you guys, you can’t do that on an mp3 now can you?”

It’s clear the vinyl offers something in the experience, and not just for DJs. As I recounted in my last post, much of it revolves around a search for authenticity, which is just the same craving that Neil Young is banking on for the success of his new Pono music player.

Pono is the culmination of the “quest” he’s been on “for a few years now, to revive the magic that has been squeezed out of digital music.” By blending the true sounds found only in vinyl with the convenience of digital music, Pono brands itself as “a grassroots movement to keep the heart of music beating.” which “aims to preserve the feeling, spirit, and emotion that the artists put in their original studio recordings.” Ultimately, with this music player, you can “hear the nuances, the soft touches, and the ends on the echo – the texture and the emotion of the music the artist worked so hard to create.”

In the FAQs, the Kickstarter campaign funding this music player directly addresses perhaps their largest market, the audiophile. In fact, the player takes special pains to remove all traces of digital sound from their digital music player.

[“The digital filter used in the PonoPlayer has minimal phase, and no unnatural (digital sounding) pre-ringing. All sounds made (including music) always have reflections and/or echoes after the initial sound. There is no sound in nature that has any echo or reflection before the sound, which is what conventional linear-phase digital filters do. This is one reason that digital sound has a reputation for sounding “unnatural” and harsh."…“The DAC (Digital-to-Analog Converter) chip being used is widely recognized in the audio and engineering community as one of the best sounding DAC chips available today.”](https://www.kickstarter.com/projects/1003614822/ponomusic-where-your-soul-rediscovers-music#project_faq_83689 " href=“http://www.fastcolabs.com/3027720/neil-youngs-crowdfunded-quest-to-fix-your-disastrous-mp3-situation">"At launch, the PonoMusic store will focus on selling albums rather than individual tracks like iTunes. Prices will v”)

The format of the music being sold in the PonoMusic store is clearly marketed toward the vinyl-aficionados as well, by the fact that “At launch, the PonoMusic store will focus on selling albums rather than individual tracks like iTunes. Prices will vary depending on record label preferences, but on average they are expected to range from $15 to $24 per album”. Vinyl sold singles back when it was the only medium for audio around, but now it trafficks largely in LPs and the occasional independently released 7-inch EP.

The Kickstarter page goes through great lengths to detail the differences between the different bit rates of digital music files available and which ones the Pono player specifically caters to, namely the FLAC (Free Lossless Audio Codec), which preserves from 1411 to 9216 kbps compared to the 192 to 256 kbps of an mp3 file. As an article about the player makes clear,

“today’s popular formats are clearly sufficient enough for the millions of music fans who purchase songs from iTunes and stream from services like Spotify. For audiophiles, lossless high-quality formats have long scratched the itch MP3s couldn’t. What Young and his team hope to do is take high-res audio up a notch and then market it to a crowd beyond the audio codec nerds.”

The conclusion the author of that article comes to is in favor of the player, recognizing that “Pono does sound different. It surfaces new things to the listener. As many have pointed out, the sound is “warm,” not unlike the analog sound of high-quality vinyl. The results will undoubtedly vary from album to album and speaker to speaker, but on the whole it does sound fuller and more pure than the audio files we’re used to” The purity of analog music is something that Young and the Pono team likens to photography, another medium subsumed by the digital revolution but which has begun to see a renaissance of sorts when it comes to film. As the Kickstarter FAQs detail,

“the way music is often listened to now, it allows us to recognize it as a song we might know, or a new song we might like, but not to hear or feel the music the way the artist created it. When we see a low quality photograph, we can almost always figure out what we are looking at - but when we study a truly stunning photograph, with the glorious detail and color and light and focus that a talented photographer can create with a great camera, we feel that picture differently – we are almost taken to the place it was shot, and to the same degree as listening to wonderfully recorded music, we can feel like we were “there”.”

This presence evoked by a truly great quality sound file or photograph is something that Pono attempts to recreate.

Photography

As Nathan Jurgenson dissects in an essay about the use of faux-vintage filters in digital photography

“For many, and especially those using faux-vintage apps, photography is primarily experienced in the digital form: snapped on a digital camera and stored and shared via digital albums on computers and websites like Facebook. But just as the rise and proliferation of the mp3 is coupled with the resurgence of vinyl, there is a similar reclaiming of the aesthetic of the physical photo. Physicality, with its weight, smell and tactile interaction, grants a significance that bits have not (yet) achieved. The quickest way to invoke nostalgia for a time past with a photograph is to invoke the properties of the physical, which is done by mimicking the ravages of time through fading, simulated film grain and scratches as well as the addition of what appears to be photo-paper or Polaroid borders around the image.”

There is a very valuable element of authenticity derived from simulating film, which can easily be extended to the use of film itself.

One particular film, Tri-X by Kodak (used by “Henri Cartier-Bresson, Garry Winogrand, Alfred Eisenstaedt, Irving Penn, Richard Avedon, Josef Koudelka and most of the finest of the photographers who worked for the Magnum agency” is valued for those authentic, yet forgiving qualities.

Using film for photography rather than relying on digital “goes to the heart of how we see and what we see and what we may be losing as billions of casual, digital snaps are taken daily and as photographic integrity is subverted by the dead, flawless, retouched faces of actors and models that gaze blankly out at us.”

A photographer revels in the imperfections of film, much like audiophiles treasure the subtle scratches and clicks of a vinyl record, rhapsodizing that ““Grain is life,” Corbijn says, “there’s all this striving for perfection with digital stuff. Striving is fine, but getting there is not great. I want a sense of the human and that is what breathes life into a picture. For me, imperfection is perfection.” Just as DJs continue to debate the merits of using vinyl or laptops to conduct their sets (though for high profile DJs like Daft Punk, there is no debate), photographers disagree about film and digital: “Film versus digital, McCullin points out, is still a debate among professionals and they are not talking about megapixels. Film is about more than just resolution, it is about authenticity. Film has other, more mysterious qualities.”

This trend has continued beyond just professional photographers, into the consumer market as well.

“In photography, the company Lomography has carved out an odd little niche market for plastic and toy film cameras whose dodgy construction produces bizarre and unpredictable effects. Polaroid cameras are on sale again and Fujifilm has produced its own version of instant snappery with the Fuji Instax film. Online, the 100m-plus users of Instagram can apply filters to give their pictures a variety of retro, filmish looks. People seem to be perversely drawn to the shortcomings of film photography—the light leaks of Lomography’s toy cameras, the strange starkness of small Polaroids and even, on Instagram, the corner-vignetting produced by vintage lenses.”

Film also sits in counterpoint to the immediacy of the digital that was decried by book readers (leading to the comeback of serialization techniques), as a photographer recounts:

“There’s nothing like going to Vietnam,” says McCullin, echoing the thought, “when I had to sit on the film for six weeks with a mental memory of the images I took. I had to be patient and carry all that film back to England. It became more precious by the week. And then you went to the lab wondering whether what you had was as good as what you thought you had. The waiting and the torment gave an edge to the whole procedure. Now, with digital cameras, you are looking at the screen on the back after every shot. It becomes an instant thing like fast food, it takes something away from the original menu.”

Film is authentic, slow, and grainy. Digital is inauthentic, immediate, and overly clean. Vinyl is authentic, grainy as well, and a medium with its own quirks. Digital music is lossy, unnatural, and reduced in most formats to a recognition of a song rather than a true representation of it. And books are an experience, spatially valuable for memory, and a medium with its own quirks, such as pop-up books, that can only exist in paper books. E-books cause words to slide off the page, neglect visual content, and scroll infinitely.

However, digital photography is booming, and the faux-vintage filters examined by Jurgenson are just as popular as the photography itself. Vinyl is making a comeback, and digital music is being remade to recapture the sounds lost in compression. Books and e-books continue to coexist, and storytelling has taken on new (old) formats of serialization that live in exclusively digital media like Twitter and apps. The cycle continues as we attempt to use digital media to recapture and reclaim the “authenticity” lost to the past. A fetishization of perhaps “inferior” media, yet imperfection is human, authenticity is found in the graininess and the grooves, and digital for now, remains a bit too far in the uncanny valley to seem authentic and real.

Women, the Web, and the App Takeover

Fri, 14 Mar 2014 20:00:12 +0000

Here’s what was important this week…

Today is Pi day. Here is more than you probably ever wanted to know about pi day.

Last Saturday, March 8 was International Women’s Day. Started as a revolutionary holiday to honor the achievements of women, International Women’s Day is recognized in many countries. However, in Nepal it is recognized by women only, rather than as a day where men pay tribute to the women. Nepal also has another holiday that only women observe:

“In early September in Nepal, Hindus – who make up 81 per cent of the country’s 30.5 million people – celebrate Rishi Panchami, a festival that commemorates a woman who was reborn as a prostitute because she didn’t follow menstrual restrictions. It is a women’s holiday, and so Nepal’s government gives all women a day off work. This is not to recognise the work done by women, but to give them the time to perform rituals that will atone for any sins they may have committed while menstruating in the previous year. (Girls who have not begun menstruating and women who have ceased to menstruate are exempt.)”

However, the interesting thing about a cultural distaste and monthly banishment that occurs surrounding menstruation, is that “they talk openly – more openly perhaps than the average teenage girl in the UK might – about what they use for sanitary protection. Some use sanitary pads, some are happy with cloths, although they dry them by hiding them under other clothes on washing lines.”

Speaking of sanitary pads and periods, this Indian inventor recently went viral again for his sanitary pad revolution… on which he gave a TED talk about in 2012 and was featured in this Fast Company article in 2011. So if you missed it all three times, go read about it.

I’d apologize to the men reading this newsletter for all this menstruation talk, except that “Menstruation is something men can learn about but never experience. Yet, our society treats periods as personal and private, something sort of gross that shouldn’t be discussed except in hushed tones or vague references.”

Just as important as sanitary and menstrual autonomy (perhaps an odd thing to call it but I’ll stick with it) is digital independence and knowledge for women. In fact, “In addition to providing a platform to share their voices, technology makes information easily available so girls and women can learn and make informed decisions.” However, “In low- and middle-income countries, 300 million fewer women have mobile phone subscriptions than men, and globally, an estimated 200 million fewer women are online. According to a report from the UN’s Broadband Commission for Digital Development, only 29 percent of women in the developing world are online.”

Although it still isn’t widely accessible globally, the Web turned 25 on Wednesday and all sorts of celebrations were had. Pew Research released a study about the web, and culminated that into 15 theses about the web compiled from experts in the field. I’ve excerpted some of those here:

Information sharing over the Internet will be so effortlessly interwoven into daily life that it will become invisible, flowing like electricity, often through machine intermediaries.

The Internet of Things, artificial intelligence, and big data will make people more aware of their world and their own behavior.

The spread of the ‘Ubernet’ will diminish the meaning of borders, and new ‘nations’ of those with shared interests may emerge and exist beyond the capacity of current nation-states to control.

The Internet will become ‘the Internets’ as access, systems, and principles are renegotiated

Those last two theses I find especially interesting, as they are nearly in opposition with each other. As Ian Peter writes about the last thesis:

““The Internet will fragment. Global connectivity will continue to exist, but through a series of separate channels controlled by a series of separate protocols. Our use of separate channels for separate applications will be necessitated by security problems, cyber policy of nations and corporations, and our continued attempts to find better ways to do things.”

This could happen through shared, local mesh networks, or through a more insidious method perpetrated by authoritarian governments that develop their own, bordered version of the web, or something else entirely!

An additional thesis that I find quite relevant considering my role in IT support:

Most people are not yet noticing the profound changes today’s communications networks are already bringing about; these networks will be even more disruptive in the future.

As Nishant Shah continues on that thesis, “It is going to systemically change our understandings of being human, being social, and being political.It is not merely a tool of enforcing existing systems; it is a structural change in the systems that we are used to. And this means that we are truly going through a paradigm shift — which is celebratory for what it brings, but it also produces great precariousness because existing structures lose meaning and valence, and hence, a new world order needs to be produced in order to accommodate for these new modes of being and operation. The greatest impact of the Internet is what we are already witnessing, but it is going to accelerate.” (emphasis mine)

A website has been established to celebrate the web’s birthday (fittingly). On it, Tim Berners-Lee, the creator of the web, lists off some of the challenges in allowing the web to continue to succeed:

How do we connect the nearly two-thirds of the planet who can’t yet access the Web?

Who has the right to collect and use our personal data, for what purpose and under what rules?

How do we create a high-performance open architecture that will run on any device, rather than fall back into proprietary alternatives?

While the web is 25, the Internet that it is built on is much older, and the infrastructure is fairly deprecated. Occasionally it takes some reminding that “The web doesn’t run on some ethereal cloud but on real physical networks which have taken considerable investment to produce.” This is especially important to remember as the Internet of Things and ubiquitous or pervasive (or whatever buzzword it is referred to as of late) computing becomes more common on the marketplace and in our everyday lives.

As the web matures, it is important to consider how we manage our identities on it. Every password and username created and (hopefully not) reused, is another potential vector for an intrusion in your digital and personal life by a criminal. I wrote a blog post about some options for improving how identity is managed online, and while it’s imperfect, I think it’s an important discussion to have.

I didn’t have a smartphone until a year and a half ago. Sometimes I wish I didn’t own one, because the keyboard is aggravating, or because I can’t quite use it right with just one hand. But I’m mostly grateful that it gives me portable access to the web. One thing that I still struggle to understand, however, is the fascination with turning everything into an app.

“Your platform now has a million apps? Amazing! Wonderful! What they don’t tell you is that 99% of them are awful junk that nobody would ever want.”

It’s all too common, and it’s become the trendy thing–turn things that are perfectly functional on the web, just for the sake of also offering an app. Often, the question of “what vehicle should this service be made available through” doesn’t get fully evaluated. The default can become the app.

Apps also allow us to enhance our lives, or as Evan Selinger puts it, to outsource our humanity. As he points out, “we need to consider the consequences of this latest batch of apps and tools that remind us to contact significant others, boost our willpower,provide us with moral guidance, and encourage us to be civil. Taken together, we’re observing the emergence of tech that doesn’t just augment our intellect and lives — but is now beginning to automate and outsource our humanity.” (you can even use an app to figure out why you’re fighting with your significant other)

I’m not anti-app, however. Many apps have a lot of utility (those that Selinger derides, even) because as with most things, it all depends on how you use them.

This next article was written by Selinger 2 years ago, and defends using apps for many things. (I didn’t even realize that he wrote it, as I tend to unintentionally ignore bylines when reading). Back then, he discussed why it’s okay to let apps make you a better person: “App designers in touch with the latest trends in behavioral modification–nudging, the quantified self, and gamification--and good old-fashioned financial incentive manipulation, are tackling weakness of will. They’re harnessing the power of payouts, cognitive biases, social networking, and biofeedback. The quantified self becomes the programmable self.”

One app that he brings up that also gets mentioned often on productivity sites is “Freedom is a productivity app that eliminates distraction for periods ranging from one minute to eight hours by disabling a computer’s capacity for networking–cutting off Facebook, Twitter, online shopping, e-mail, instant messaging, et cetera. That’s right, freedom now means the willful use of technology to limit one’s options!”

I use a similar app on my Macbook called SelfControl. Apps and extensions and software in general I find to be most useful when they enhance your process flow or life in some way. Many apps are focused on making your life as efficient as possible, or as a popular LifeHacker series focuses on, what you use to help you work (not necessarily apps but they come up often). Of course, the apps I use most often are messaging-based, and those are also very popular as I’ve previously discussed here.

Apps can get a lot wrong, too, as the sector creating apps isn’t representative of the population. Mike Lavigne calls fellow app designers to task when it comes to designing apps for women: “It’s time we designers stop pandering to cultural norms, start disassembling our stereotypes, and get in touch with how people–who have a huge amount of variability–actually feel about themselves.”

The latest app being hyped is called Spritz, a speed-reading app. I value skimming and deep reading (and re-reading) when I’m reading on the web. There are several downsides to speed-reading, such as a probable diminished comprehension and retention, as well as increased load on working memory, as this Atlantic article details. “But maybe apps like Spritz (there will doubtless be copycats) aren’t meant for Ulysses. That is, they’re not for reading; they’re for obligatory information processing—all the digital chaff we sort through with each glance at the RSS reader or inbox. And maybe that’s why we really do “OMG need” things like Spritz these days. Not for the joy of reading, but for the duty.”

The music this week is a pop classic, at least of the last year or so: Pitbull, ft. Ke$ha with their song Timber.

I chose to feature this song as well, because Grantland published a profile on Kesha recently (and her rebranding away from Ke-dolla-ha), and it’s really worth the read.

If Kesha isn’t your style, here is Wolf Alice with Leaving You for a bit more of a folk-blues-rock style?

Or you can try Josef Salvat with his song Every Night, for a groove feel.

Thanks, as always, for reading!

Identity on the Internet

Sun, 09 Mar 2014 21:22:03 +0000

Anonymity is valuable to the structure of the Internet, but as the identity of a person becomes fluid, the reputations and identifiability of someone’s online presence becomes increasingly valuable. While jobs rely on user-submitted references, as do academic applications, many also turn to your social media presence or to your search results to gauge reputation. Privacy by obscurity, as records are digitized and indexed, is no longer as viable. But, there is no consistent form of identification across the web. Each service relies on its own username as identifier, with character limits abound, and your ability to hold the same username across services relies on both the uniqueness of your username as well as the date you joined the online service. But are usernames outdated? A self-selected identifier, varying from service to service and format to format? As Mat Honan puts it, "“One of the best things about the online world is how it lets us be whoever we want to be. We shouldn’t have to sacrifice that just because someone else got there first."

The advantage of a username is that, at least within a service, it “refers unambiguously to a particular person”. That works fine if you know the username of the person, but often you may only know their name. Luckily, with services like Facebook, a person’s unique identifier is their name, provided they haven’t pseudonymized it. Once you have connected with that person, you expect (within the relevant online service) when you type in their name, you will be returned with precisely the person you were expecting to find. The difficulty with this system is finding out the username of another person, and confirming that the person with their name online is really the person you’re looking for.

Names and usernames online don’t always align with the identity of the person you expect them to be tied to. To link an identifier such as a username to a person, you must gather attributes about the person behind the username to confirm them against the facts that you know about the person you’re trying to locate. You assess the photos, mutual friends, and locations of the Facebook search results for John Smith to find your friend. You conduct the same operation when you look someone up in a phone book, confirming that the John Smith in the phone book is the same John Smith that you are looking for, because he lives on Alphabet Street.

Online identities allow us to be semi-anonymous through this need of attribute verification, but there is a limit to any anonymity granted online. As privacy by obscurity fades in capability as records are digitized and indexed, many are choosing to reclaim their names as identifiers, to build their own reputation instead of the one the internet builds for them. We select usernames as an extension of how we choose to be identified. Pseudonyms, subtle homages to the things we care about, our nicknames, and eventually, some variation on our actual names. Whitney Erin Boesel describes this transition well in Cyborgology, concluding with her decision to change her username on Twitter to a variation on her name. “Identity online should take a cue from the physical world. You are more than your name; your face, your birthday, your location, and the company you keep all help others figure out who you are.”

As our personal identity becomes increasingly fluid–less tied to our locations and our jobs, and more tied to our self-representation on the internet, how we are identified is immensely important:

“For many of these people, their Twitter account or Facebook page is their identity. It’s the place where they present themselves to the world. These sites have taken off partly because our other identities have weakened – or, as the sociologist Zygmunt Bauman puts it, have become “liquid”. People once defined themselves by their job, church, nation and family. But in these secular, jobless, globalised times when ever more of us live alone, we are no longer very sure who we are.”

Digital identity is a valuable and curated thing. Nearly a brand, and as one becomes more famous or well-known, it must be protected against impersonators As Choire points out, even governmentally the only proof we have that someone is who they say they are is their Social Security Number:

"“How do we know who people are? We have some definite if hackable systems, like social security numbers. Names are a problem; sometimes unique, often not. So people are who they say they are—except, more and more, they are who we’re told they are.””.

Many people who I spoke with alternated between using their real names and pseudonyms based on the service–mostly due to privacy reasons. Pseudonyms based on site context (dating, personal social media), and also to add an extra layer of “discovery fatigue” so to speak–if you make it harder to find, fewer people will try. Others use their real name, because it keeps them honest or because it’s “what I got to trade on”. It is difficult to craft a reputation for your online identity if no one knows who you are.

Most often, a username is more pseudonymous yet transparent than a name, as it allows the machine to be sure that it has located precisely the requested person. A username functions as a publicly known unique identifier for the databases that house our information:

“our data has to attach to unique identifiers to live on a server, but only the machines need to see those. They’re just like the Social Security numbers we use in meatspace to differentiate people with the same name. Ultimately we’re all just numbers to computers anyway. It’s kind of counterintuitive, but the best way to be whoever you want to be is to be nothing more than a number to everyone but your friends.”

Numbers are one way to bridge the information gap–no attributes to verify about someone if they always align with the same string of numbers. So perhaps it’s time to turn to biometrics as the definitive source of online identity. After all, while biometrics are widely touted as a good replacement for a password, once a biometric password is stolen, it can’t be changed. Your retinal makeup or fingerprints cannot be (easily) altered. Thus, biometrics would work far better for usernames, with a large caveat being that anonymity would be difficult (if not impossible) to maintain. How could one avoid the tying together of work and personal and private social accounts, if they are all linked by your thumbprint?

Fortunately, “Biometrics can be use used as a lightweight, convenient mechanism to establish identity, but they cannot authenticate a person or a thing alone.” This is true because fingerprints are not private information, like passwords are (or at least should be). Fingerprints are known to the government and something that we leave all around us, whenever we touch something. Much like our names and usernames, they are uniquely identifiable as belonging to us, but without requiring a second layer verification of attributes. Match John Smith’s fingerprints and you no longer wonder if he is the one that lives on Alphabet street.

Using fingerprints or other forms of biometrics as online identifiers would also work on the computer side, as this data is encoded as a unique set of identifiers, as well as on the user side, with a simple thumb or fingerprint and a password (+ two factor token) easily used to log in to a service.

Of course, biometrics are more difficult to use on the social web. How do you know that the person you’re emailing is your friend? You don’t have their fingerprint. Perhaps they would be used in tandem with usernames, voluntarily. To tie a username to my “true identity” I would add a layer of biometric identification to lend authenticity and verification of my identity to the service. However, for social media endeavors where I might prefer to remain semi-anonymous, I could leave that service un-authenticated–preferably through an opt-out box that would prevent others from authenticating themselves against my own service.

This sort of relative simplicity could avoid some of the protracted and desperate steps that others have had to take, such as selecting a name for their child based on domain name availability, and then registering that domain name and a variety of social media services based on that future name. No one could impersonate you without your fingerprint credentials, password, and tokencode.

While you may not be able to choose the username that you wanted, your friends would always know that they were contacting the correct version of you online, as the fingerprint of that service would match your own. An issue with this is that once a harasser or stalker has access to your username or fingerprint, they could have access to all of your verified accounts. Of course, using a single point of verification across many services could also allow blocking of someone else’s fingerprint from accessing anything associated with yours. You would then be able to ensure relative privacy across your social media universe.

The idea of using biometrics as the unique identifier behind your digital identity aligns with public key cryptography, which uses the PGP system. PGP, or pretty good privacy, uses a combination of a public key (in my example, your biometric identifier) and a private key that only the person receiving the message holds (in this case, a password or something of the sort that only you know). This is also combined with a session key that is tied to the particular message itself. If “we can bring the idea of cryptography like PGP to a user’s identity – this is a verified way of proving I am me – if that was bound indefinitely to me, that offers a tremendous amount of value to individuals and companies.””

Online identity protocols which use unique identifiers to identify each user, provided by identity providers (IdPs) ranging from social media services (Google, Facebook, Yahoo, Twitter) to educational organizations (InCommon, other higher education institutions) that verify the identity of the user, and allow a trusted connection between the identity provider and the service that the user is attempting to access. In OpenID the user is typically identified by a URL, but that doesn’t work that well because [“ordinary human beings apparently don’t think of themselves as Resources that need Uniform Identifiers, or some such.”](https://www.tbray.org/ongoing/When/201x/2014/03/01/OpenID-Connect " href=“https://www.tbray.org/ongoing/When/201x/2014/03/01/OpenID-Connect""). OpenID Connect, a newer identity protocol, uses email addresses–which can, again, provide many of the same issues that we’ve discussed about false identity. This would also likely entail using whitelists, wherein “apps will explicitly specify which IDPs they’re interested in relying on.” so as to provide some sort of consistency and verification as to who is accessing the service.

Each of those services relies on walled gardens of identity which keep us cloistered in each service. Cross-platform and cross-service interactions are incredibly more valuable than individual services, each of which require their own sign-on or a central IdP to grant sign-on privileges to the service providers which allow it (such as with social ids. Why can we not sow our own gardens, and be our own IdPs? The information being provided by each IdP would need to be consistent and able to be consumed by the service providers, and with biometric identifiers that would be possible. Have different encodings for different kinds of identifiers, and each person could use whichever one they preferred or were able to use.

To further allow for pseudonymity, Stoya comments that:

"“Maybe it would be easier to navigate the dissolving boundaries between public and private spaces if we all had a variety of names with which to signal the aspects of ourselves currently on display. And maybe we should remember that our first glimpse of a person is just one small piece of who they really are.””.

There is a necessary incompleteness to online identity, and yet while the relative completeness is controlled by ourselves, the identifications associated are not. A localized and personally owned identity provider with a relatively static identifier such as a thumbprint (with concessions made for those without thumbs or which choose to use other biometric identifiers) would still allow us to select our own usernames. But further, we’d be granted the permission to choose which of our own attributes to share with certain services–even none, if we want to verify our identity yet continue to remain anonymous. Your online identity is not your whole self, but what is shared and what is revealed to others should be up to you to release.

Edit 3/11/14: A friend told me about the new site https://keybase.io/ which achieves much of what I discuss in this post, most notably the ability to sign or verify your identities as being owned by you (and not impersonators). Additionally, it makes PGP encryption relatively simple and easy to use for the average person. They’re accepting requests to join their beta list. It will certainly be interesting to see how well and how widely cryptography and encryption can and will spread in the mainstream. My guess is that either some very large companies will need to start doing it by default, or it privacy will remain a luxury good--available only to those with both the time and money needed to invest in setting up these processes.

Bitcoin, Security, and Photography

Fri, 07 Mar 2014 21:00:19 +0000

nananananananananananana BITCOINNNN

I had to talk about it eventually, and Thursday’s news was a good impetus. Newsweek had a big “scoop” potentially unmasking the founder of Bitcoin. The magazine saved this story for the cover of their return-to-print issue. The story features stalking masquerading as investigative journalism, as the author tracked down this man through national records, then tracked his interests to a model train forum, where she emailed him purporting to be interested in trains, then began asking about Bitcoin (at which point he stopped responding). Then she tracked down his home and family members, and interviewed them extensively about the man and itcoin. She finally paid him a visit at his home, and instead of answering the door he called the cops. This surprised her. Read the article in full, if you’d like to know more about the lengths some people will go to find people who don’t want to be found (and who haven’t done anything wrong).(After some sushi and a car chase the man himself claims he is not involved with Bitcoin).

Don’t know anything about Bitcoin? That’s okay! It’s a cryptocurrency that’s difficult for me to easily explain, but this essay does an excellent job at it, beginning by answering two questions:

1) Should I buy Bitcoins?

No.

2) But I keep seeing all this stuff in the news about them and how

No. Tech journalism is uniformly terrible, always remember this.

Bitcoin has numerous issues in my opinion, all covered neatly by that article, with passages that point out:

“the Bitcoin network now must use vast amounts of power just to maintain itself, power typically generated by fossil fuel plants and in amounts far out of proportion to its actual usefulness. It is a tremendous waste of actual real-world resources that could be better used on something important “

And additionally, subverting the hopes and dreams of technolibertarians everywhere:

“If Bitcoin actually became popular as a currency and not just as a speculative commodity, the blockchain would swell to an absurd and unmanageable size. Visa (for example) maintains multi-terabyte (at least) databases of financial transactions; now imagine if everybody who wanted to safely use a Visa card had to have a copy of all that data (including lists of everybody else’s transactions).”

Businessweek also details a bit more about the damaging aspects of Bitcoin and Bitcoin mining. At least if Bitcoin crashes or if eventually too many Bitcoins are stolen (like with exchanges Mt Gox and Poloniex and the Bitcoin bank Flexcoin) we’ll have some pretty sweet computer gear. A lot of people were attracted to the secrecy and anonymity of Bitcoin, but they may be realizing that they were investing real money into something easily stolen, without anyone standing by to insure your money is returned (thanks, FDIC!) We’ll see how the popularity of Bitcoin is affected by these revelations, or if new Bitcoin exchanges will step up claiming to be more secure than their predecessors, and the cryptocurrency will live on!

Using secure devices and tools that help protect your privacy is always a good idea. However, many people don’t use them because they’re not easy to use (or because they believe the fallacy that they have nothing to hide so it doesn’t matter). Now, companies are starting to realize the importance of user experience and good design when implementing popular and useful security tools for the masses. Writing about a new secure smartphone called Blackphone, John Pavlus remarked “The value of obscuring my personal communications from surveillance just didn’t seem tangible.” But the phone is attempting to change that, as the Blackphone’s “apps are intended to look and feel just like the normal phone, texting, and address book apps you’re already used to using, so whenever possible the user doesn’t need to learn any new skills or complex steps in order to reap the benefits of privacy." (Speaking of smartphone security, beware of malware like this on your android device)

Changes are happening in other realms beyond smartphone development. Social media efforts are being made, for example by some students who:

“built a new interface for community-sourcing oral history and family photos and documents. They built privacy questions into the site so, for example, every photograph uploaded required documentation of who was in the photo and what level of permission had been given to display it. You could not post the photo without addressing all the questions—questions that themselves had arisen from a community-based discussion about privacy.”

By putting the emphasis on privacy rather than ease-of-sharing, the risks of sharing on the internet are made transparent. For example, “if the goal is to save people from the over-centralized riskiness of Facebook and Gmail, you’re not going to get anywhere without offering a user experience that is at least comparable to those shiny, accessible services.”

Open source software efforts aside, much of security and privacy-focused efforts seem to be available only to those with the money to buy them or the time to spend configuring and using them. Indeed, it may be that “In our data-saturated economy, privacy is becoming a luxury good. After all, as the saying goes, if you aren’t paying for the product, you are the product. And currently, we aren’t paying for very much of our technology.” Silence is also becoming somewhat of a luxury, as “not only do we value it in a general sense, we’re willing to pay for it.” Technology has a large part to play in this change, as it “has both increased our perceived need for silence and created (or at least improved) the means of attaining it. We’re assaulted by incessant technological “noise” and reliant on technology to control it.“

Reading tech journalism is stressful, weird, and insular at times. I don’t recommend it too much, even if it is still interesting to me. Let’s talk about photography instead.

Getty Images, one of the largest image licensing entities in America, recently changed tactics and rather than sue people who used unlicensed versions of their photos, are now making 35 million of those photos available as embeds. Nieman Lab goes into more detail about how to use them and how they will work on the web. Previously Getty Images has been in the news after partnering with Sheryl Sandberg’s Lean In organization to provide more “representative” stock photos of women, as well as when they partnered with social media site Pinterest to provide more context for unsourced images.

Photography as a medium, however, is far more interesting to explore. John Quincy Adams was the first president to be photographed, and his reactions upon viewing the photographs are reflective on the value of photography:

“In a diary entry for Aug. 1, 1843, Adams noted that four daguerreotypes had been taken and pronounced them “all hideous.” Three more were taken the following day, but he found them “no better than those of yesterday. They are all too true to the original.””

Photography is typically valued now due to its faithfulness to the subject, perceived as a medium which lessens the distortion of the situation. But John Quincy Adams preferred his painted presidential portrait, and as Greg Ross continues,

“Which of these images is the more revealing record of the man? In Puzzles About Art, philosopher Matthew Lipman asks, “Which would we rather have, a portrait of Socrates by Rembrandt or a photograph of Socrates?””

Does the near-exact likeness produced by a photo indeed capture the “true representation” of a person? Or is there more to be gained with the finesse, creativity, and interpretation involved in other artistic media?

Choosing whether or not to take a photograph at all, is a difficult decision. Julia Phillips, an essayist who followed along on a sled dog race in Siberia was faced with this reality after witnessing the wildness of the dogs as they attacked. As she recounts, “I lifted my camera to take a picture. “Don’t,” she said. “Who needs to see this?””

What should be recorded, and what should be left silent to memory and history?

Another form of photography is film–the moving picture. Sergei Eisenstein pioneered the technique of montage in film, most notably through a depiction of a (fictitious) massacre on the steps of Odessa. The Kuleshov effect further reveals the power of montage: “An audience reacts not to a film’s elements but to their juxtaposition — the sequence of images suggests an emotion to them, and they project this onto the actors.”

While Eisenstein’s most famous scene was staged for the purposes of the film (a Soviet propaganda film), Dziga Vertov is most renowned for his documentary filmmaking style, recording life through the interpretation of the camera. You can watch the entirety of his most famous film, Man with a Movie Camera, on Youtube.

Finally, to close out this week’s newsletter, here is a song about bicycles by Rob Cantor.

As always, thanks for reading!

Writing and Race

Fri, 28 Feb 2014 21:00:56 +0000

Here’s what was important this week…

I spend a lot of time writing, but it never seems like enough. Partially because I spend so much time reading the writing of others, and partially because a lot of the writing that I do is IT documentation for my job. I feel truly accomplished when I manage to finish a blog post (there are at least 11 partially completed, with an entire doc full of more ideas). A lot of the time that I spend working toward a blog post is spent reading, tweeting, and tumbling (how I archive the articles I read). I tell myself it’s like research, and I do find it to be valuable network-building especially when I find a rich creative environment lacking at times. Writer Emily Gould told herself many of the same things, until she had a realization:

“For many years I have been spending a lot of time on the internet. In fact, I can’t really remember anything else I did in 2010. I tumbld, I tweeted, and I scrolled. This didn’t earn me any money but it felt like work. I justified my habits to myself in various ways. I was building my brand. Blogging was a creative act—even “curating” by reblogging someone else’s post was a creative act, if you squinted.”

She was trying to write a book, but only spent time on the internet. (Jacobin has more on the literal labor of social networks online).

Another writer finds a similar issue in a different way. As Neil Gaiman puts it to Edward Nawotka, “Use your blog to connect. Use it as you. Don’t ‘network’ or ‘promote.’ Just talk,” He goes on to clarify, “It’s much more important to write than to blog, so only blog if it makes you happy and if you have something to say,”

I’ve enjoyed writing for a long time, but never really thought I would have a job where most of what I do is write. I suppose one clue could’ve been an ongoing fascination with words and language. When I was younger I would flip through the dictionary and find words that caught my eye, like the idiosyncrasy between the main definitions of sanguine and sanguinary. Alexis Madrigal, senior editor at The Atlantic, finds a similar interest. As he details in an interview about his own newsletter:

“I love how people talk, strange and slangy. I love when people make up words. An old friend’s father was Persian and used to just drop random the syllable ‘anj’ onto the ends of words when he liked how it sounded. You always knew what he meant, but the rhythm and feel changed: candelanj, afaranj, timeanj. I love when rappers play with the language and bend it into shapes you didn’t think were possible. I’m for a mongrel language, a people’s language, a perpetually created and destroyed language.”

As I’ve written about before in an earlier newsletter, the idea that there is only one way to interpret language leads to an enforcement of literary privilege, and this can extend easily into racism against those that can’t “speak English right”. As observed by Lynne Murphy writing for the Oxford Dictionaries blog, this interpretation is primarily American, because we ascribe so much meaning and prescriptive power to the dictionary that we view word definitions as key to court decisions, unlike the British. (This holds at least for English, there may be other languages/cultures that do the same).

The alternative, of course, is to encourage bilingualism. Canada is has two official languages, and thus enforces bilingualism–and recently extended that enforcement to social media posts. Bilingualism is more common, however, in families in which the parents are from a country but are raising their children in another, or when the parents are from different countries and speak different languages. Ben Faccini writes about the challenge of attempting to raise bilingual children after growing up with a French father and an English mother.

Ta-Nehisi Coates recently learned French, and recognized the value of knowing another language.

“It turns out that means more than talking to people, reading books or watching movies. It means understanding the difference between a definite and an indefinite article, the deeper meaning behind “Prêt A Manger” or “Le Pain Quotidien,” or the fact that the language you take as foreign is actually “everywhere”—on the buses and trains, on the lips of mothers remanding children, out the mouths of cab-drivers yelling at each other. “

His recognition that a language does not seem to be present until you learn to speak it parallels his further comments about how lists of the greats, “like “Geniuses of Western Music” [were] written by people who evidently believed Louis Armstrong and Aretha Franklin did not exist.” because they did not bother to learn about more than was familiar to them, they ignored the accomplishments and contributions of such phenomenally successful artists as Louis Armstrong and Aretha Franklin. As he concludes, “Here is the machinery of racism—the privilege of being oblivious to questions, of never having to grapple with the everywhere; the right of false naming; the right to claim that the lakes, trees, and mountains of our world do not exist; the right to insult our intelligence with your ignorance.”

The erasure of people of color, and an emphasis on the value of being white, was so strong that some people went so far as to move across the country (usually at the insistence of family members) and pass as white in order to find more success. A Stanford professor is researching their experiences, and points out that she is “not as interested in what people gained by being white, but rather in what they lost by not being black." Racial identity is a strong and valuable element of one’s personal identity. The college newspaper at University of Michigan, perhaps stirred by the controversy of the #BBUM movement, began a daily opinion column called Michigan in Color where people of color can write about their life experiences. They are adding voice to a traditionally voiceless minority. Some of the best (all short, I promise) are:

Even powerful people of color, like Kanye West, struggle with this. But as he points out in an interview with director Steve McQueen:

“Too many people are scared. But it is my job to go up every night and talk about this kind of shit. It is actually my job. I’m like a broadcaster for futurism, for dreamers, for people who believe in themselves. We’ve been taught since day one to stop believing in our own dreams. We’ve had the confidence beaten out of us since day one, and then sold back to us through branding and diamond rings and songs and melodies—through these lines that we have to walk inside of so as to not break the uniform or look silly or be laughed at. So I hope that there are people out there laughing. Laugh loud, please. Laugh until your lungs give out because I will have the last laugh.”

When other people define your experience, through stereotypes or microaggressions, it can have a serious effect on your life. As one man recounts in an excellent long essay about mercy:

“I’ve had to struggle not to absorb those stares and questions and traffic stops and newscasts and tv shows and movies and what they imply. I’ve been afraid walking through the alarm gate at the store that maybe something’s fallen into my pockets, or that I’ve unconsciously stuffed something in them; I’ve felt panic that the light-skinned black man who mugged our elderly former neighbors was actually me, and I worried that my parents, with whom I watched the newscast, suspected the same; and nearly every time I’ve been pulled over, I’ve prayed there were no drugs in my car, despite the fact that I don’t use drugs; I don’t even smoke pot. That’s to say, the story I have all my life heard about black people — criminal, criminal, criminal — I have started to suspect of myself."

Typically, the best place for people to learn about race and racism is in school. But as Tressie McMillan Cottom points out, “When colleges and universities become a market, there is no incentive to teach what customers would rather not know. “ And the majority of customers, being white, would rather not learn about race. As she continues:

“A white student may feel discomfort when it’s pointed out to him how he has benefited from structural racism, but to compare that discomfort to discrimination is a false equivalency. Hurt feelings hurt, but it is not oppression.

But hurt feelings can be bad for business. And a lot of powerful people think colleges should act more like businesses. When they do, students act more like customers. And our likely customers might not be amicable to discussions about structural racism. If the customer is always right, then the majority share of customers is more right than the minority.”

This is a clear trend in higher education, student as customer, higher education as business or even country club.

An Inside Higher Ed article from yesterday asserts, “We are undeniably in an era where the governing model of education is one that conceives of students as customers.”

Thinking about what (if anything) can be done to reinvigorate higher education makes me a bit despondent, so let’s turn right to the music. The Vaccines have a great song that I somehow only found yesterday called Norgaard.

Give it a listen, it makes me want to call it surf punk but who really understands genres anymore. A calmer version of surf punk would be surf-wop. I’m just going to call The Drums that because why not.

A bonus song because I’ve been listening to oldies for most of the formation of this newsletter is going to continue my mini-nonexistent-theme of surfing. For the more sedentary among us, here is Otis Redding with Sitting on the Dock of the Bay.

Feel free to comment with your favorite oldie/doo-wop/soul song, and/or your theories for fixing higher education. As always, thanks for reading!

Protest and Media

Fri, 21 Feb 2014 23:15:25 +0000

Here’s what was important this week…

Are women being [infantilized](link removed) or [endangered](link removed) in the Olympics?

Also, the Olympic medal count gets more interesting depending on whether you look at it in terms of [total medals](broken link removed), [number of gold medals](link removed), or [medals per capita](broken link removed).

In world news, protests in Ukraine that have been [going on for a few months](link removed) have escalated as the government ramps up its violent response. Just today (overnight for us in the US time zone) a deal was signed between the government and the protestors. Hopefully it will hold. That article (CNN) provides a good overview of the violence, but essentially the protests started as the government aligned itself with Russia, while many citizens wished for more of an EU alignment. Photos (some graphic) of the violence were collected yesterday by In Focus, and the New Yorker is wondering if this protest is the final straw: Will Ukraine Break Apart? Like many of the protests in recent years, the protests have been named somewhat with the square in which they’re occurring. Tahrir, Zucotti, Gezi, and now the Ukrainian protests, combining the word for “square” and the crux of the protests, european integration, to make euromaidan. You can [watch four simultaneous live feeds of the park](link removed) if you like. (The current president of Ukraine also ran for president in 2004 and was “elected” but forced to concede to his opponent after accusations of electoral fraud. One of those protesting the election results also happened to be the sign language interpreter for the state run news channel.)

South America has also been beset by protests lately as well. Brazil has had ongoing protests, and they’ve been intensifying as the World Cup nears. ESPN investigated the protests this past summer in Generation June:

“The protests that took place during the Confederations Cup in June, and that you might see again during the World Cup next year, aren’t against a soccer tournament. They are against the entire structure of a society, against the issue at the core of so many other problems, and this fact alone almost guarantees a swift and violent repression.”

Venezuela has only more recently come into the public eye with anti-government protests, in part sparked by a inadequate government response to the shooting of a beauty queen(revered as celebrities in Venezuela). The [BBC has more background on the current situation](broken link removed) and about how the protests began.

One way that protests and dissent used to spread, in the years before the internet, was through media like radio, video, and music. A documentary in the making was featured in the NYTimes, focusing on the illicit dubbing of American movies during Ceausescu’s dictatorship in communist Romania. Western influence of any kind was banned and guarded against, but these movies allowed an escapist form of subversion, that gave dreams to oppressed people. Music was another way that dissent could spread, and also provided an escapism for many citizens. I wrote a paper in college about how 2 singer songwriters in Khrushchev’s Soviet Russia gave back the “everyday” experience to the people, after years of idealistic Socialist Realism. A book that got me into this concept is now sadly out of print, but focuses on the impact of a radio station in Serbia that played a crucial part in helping to take down the rule ofMilosevic. I can loan you my copy if you’re interested, or you can use that link to buy one through Amazon.

Radio is severely important in my opinion. After growing up listening to the college radio station in my hometown, and reading that book, I’m convinced that radio is an important, though neglected, transmission medium. (It’s probably no surprise that I was a DJ and worked for mycollege radio station.) The FCC also thinks radio is important, especially the fading medium of AM radio. Nostalgized in the Everclear song AM Radio, the FCC commissioner establishes:

“If you care about diversity, you should care about AM radio. Most minority-owned radio stations are located in the AM band … If you care about localism, you should care about AM radio. Many AM radio stations cover local news, weather, and community events.”

As the article explains, one of the reasons that AM is struggling is because the frequency band that it operates on is rife with interference from common household devices.

In fact, interference and human-generated frequencies are difficult to escape nearly anywhere on earth. The places on earth without any noise are vanishing. Jason Kottke discovered that when Wolfram Alpha calculates how many births and deaths occur per day on earth, it produces a result in Hz, the unit of frequency. Literally, the frequency of humanity. Orion Magazine tweeted about the frequency of urban environments in the US and Europe:

And, conveniently, Randall Munroe published an XKCD comic that compares the frequency of events occurring across the world (not in Hz). It seems that someone gets married just as often as someone registers a domain (or very nearly).

While audio media were very popular in unseating communist dictatorships, they’ve been less popular on the web, rarely if ever achieving the viral status that videos, gifs, and memes have enjoyed. Part of that reason is that there isn’t a definitive source for audio on the web. As Digg puts it in their investigation into Why Audio Never Goes Viral:

“There is no Google Sound, no BuzzFeed for audio, no obvious equivalent of Gangnam Style, Grumpy Cat or Doge. If you define “viral” as popularity achieved through social sharing, and audio as sound other than music, even radio stations’ most viral content isn’t audio — it’s video.”

However, the article does recognize my favorite source for audio on the web, SoundCloud:

“Hoping to make sound as sharable as video, SoundCloud delivers this content via a streaming player instead of a dressed-up file download.”

It has the benefit of giving you something to look at if you choose to, while also being low on processing and memory resources (especially compared to a video) so it is easy to play while doing other things. Of course, audio (and radio) allows the listener to create their own scene, and as a former radio broadcaster remarks in the same article, “What’s so beautiful about radio is you can’t compete with what people are imagining in their heads, right?”

Viral content mill Upworthy shares audio at times, like [this recent Storycorps story](broken link removed), but it is shared as audio against a black screen on YouTube—a video-based, not audio-based medium.

And as the Digg article makes clear, most audio content that has gone viral is public radio content, voicemails made public, or portions of podcasts–it’s more rare for audio to be intentionally be produced and shared in short, easily shared and digestible segments.

After reading this piece, NPR experimented with producing audio content with the intent of making it go viral, with moderately successful results. Many of the examples in the Digg piece are examples of public radio excerpts, and spoken words. However, the experiment results highlighted in the NPR piece were almost all sound based—what it sounds like inside a category 5 hurricane, what a volcano’s “primal scream” sounds like.

As Dr Robin James explores in a Cyborgology piece in response to an article on Vice, people want the opportunity to share the sounds around them with others, but the few services that offer the capability aren’t widely used. She concludes with a question, “is this due to the nature of sound and/or photography, or is this due to the nature of social media, as a type of sociality, as a media platform, and as a business model?”

I’m not sure how much I agree with her, and I’d propose that the lack of adoption in audio social media is attention. As the Digg article points out, most audio content (such as radio, and I’d extend that to podcasts and audiobooks) is consumed while doing something else (most often driving). Your car doesn’t facilitate sharing, but too, we aren’t consuming audio when there are other options enticing our attention–like easily skimmable text, or video that manages to capture at least 2 senses (visual and auditory) and can attempt to reduce distraction that way. The virality of audio, and its prospects as the content on a social media service, is not only hurt by what’s available to share (or how it’s shared) but ultimately hurt by the distracted nature of web users. NPR’s experiment is promising, however, and with enough quality content, those distracted users may be convinced to listen to a minute or two of quality audio.

When it comes to social media, formatting aside, messaging dominates. As Ben Thompson describes, “While the home telephone enabled real-time communication, and the web passive communication, messaging enables constant communication. Conversations are never ending, and friends come and go at a pace dictated not by physicality, but rather by attention.” One reason that messaging is so important for companies is that it allows them to latch on to part of that attention and use it to sell you something. Email is one of the dominant methods of that still, but they’re easier to delete without reading. Messages between friends, however, accompanied by ads for your favorite clothing store, would be harder to avoid. One messaging app, WhatsApp, doesn’t use advertising.

And as many of you may know, Facebook bought WhatsApp for 19 billion dollars in a combination of cash and stock. If you’re like me and didn’t know anything about WhatsApp before this week, The app offers [“basic messaging” and in addition, “WhatsApp users can create groups, send each other unlimited images, video and audio media messages.”](broken link removed) They also have recently introduced [voice messaging](broken link removed). Xeni Jardin explores many of the potential reasons for the purchase in her piece for Boing Boing, but they center around both the large number of monthly active users ([450 million](broken link removed)), and the app’s popularity internationally.

However, visual messaging apps are also popular messaging apps. Snapchat one of the [larger](broken link removed)(and is [remaining independent](broken link removed)), but more are trying to compete. One of these new competitors, Pop, explains the purpose behind their app:

“Much of our visual history now exists online. Pop makes this media accessible as short clips for anyone to use to express themselves. Social media has largely been about sharing what you see—taking a photo of what’s in front of you. By making it easy to combine your own recordings with any media on the web, Pop makes it possible to share what you see and how you feel about it.”

Using visual social media in general has become ordinary, with Pinterest and Instagram, among other sites and apps, dominating visual social media. The [NYTimes featured this trend last summer in a blog post](broken link removed). As photographer Craig Mod realizes, there is a definite appeal to sharing visual experiences, and it makes him rethink his own camera usage while on a 6 day hike in Japan:

“smartphones further squish the full spectrum of photographic storytelling: capture, edit, collate, share, and respond. I saw more and shot more, and returned from the forest with a record of both the small details—light and texture and snippets of life—and the conversations that floated around them on my social networks.”

As he realizes, “the data quality of stand-alone cameras, no matter how vast their bounty of pixels, seems strangely impoverished. They no longer capture the whole picture.”

While mobile becomes more popular, tablets are suffering. Re/code declares it outright: “our love affair with the tablet is over.” Their reasoning is simple:

“As phone apps improved in terms of quality and speed, users abandoned their tablets for the device in their pocket that could access the Web anywhere and anytime from Wi-Fi or cellular connections.”

However, Ben Thompson realizes that tablets have benefits compared to the traditional desktop computer, because “A Mac or PC is a superior experience for traditional computing activities, at least according to traditional measurements like speed or efficiency, but an iPad is simpler and more approachable, and it does other things as well.”

Robert McGinley Myers finds that a smartphone allows him to do so many things at once that he often finds himself overwhelmed. As he elaborates:

“I could use it to check my email, Twitter, Facebook, my RSS reader, my Instapaper queue, Flipboard, Tumblr, Instagram. I could also add an item to my todo list, write a journal entry, write a blog post, take a picture, listen to a podcast, read a book. And just as the device can be many things, so it reminds me that I can be many things: an employee, a teacher, a spouse, a friend, a family member, a reader, a photographer, a writer. I can feel it pulsing with obligations in my pocket. I sometimes find myself flipping through apps, and potential identities, the way I used to flip through TV channels.”

Christopher Butler agrees that smartphones are capable of a lot, and calls it “the first successful synthesis of three previously distinct machines: the telephone, the television, and the personal computer.”

This week’s music is a bit off the wall: The singer from Soft Cell singing along with a cover of Tainted Love…as performed by floppy disk drives.

For those of you that want to escape the technology that flooded the last part of this newsletter (and those of you that didn’t), here is Kalle Mattson performing An American Dream.

Thanks for reading!

A lot of what I find on the web I’ve discovered through excellent blogs like Boing Boing and Kottke.org, both of whom I linked to in this newsletter. Another one of my favorite blogs is Futility Closet, full of great odds and ends that he discovers in his own research. If you have a favorite blog, I’d love to hear about it! I sometimes find that I’m drowning in the tech sector news.

I also want to point out that I don’t use WhatsApp, Pop, and only started using Snapchat last weekend. I forget that I have Instagram on my phone, and forget that I have a Pinterest account. I traffic mainly in links rather than visuals, words more than pictures. But that’s just what I prefer. Clearly, a lot of people are different, and there are always some things better said with a picture. I’ll have more on photography and the power of visual images in a future newsletter. WhatsApp does seem like a practical app, however (I use something somewhat similar called Kakaotalk)(emoji are my greatest use of visual messaging), and especially given the worldwide popularity of the app (450 million active users) it says a lot about the ways that people are using the internet to communicate, but also to subvert restrictive telecommunications providers, limiting minutes and text message amounts, these messaging apps use data and/or wi-fi to accomplish the same tasks, only with more features.

Identity, amplification, and ownership on the Internet

Fri, 14 Feb 2014 17:02:35 +0000

Here’s what was important this week…

Facebook now allows you to choose a “custom” gender option and fill in your own gender on your profile–to a point. Rather than being a free-text field, Facebook instead offers options which autocomplete. Slate went through the effort of tabulating all 58 of them. Facebook is likely avoiding a free-text field because it wants to avoid trolling, but more likely they want to maintain the purity of their data about users.

One issue with Facebook (and in my opinion, this could be extended to many other social networks) is that it requires code switching. Code switching, typically associated with race and ethnicity, is even featured in an NPR blog devoted to the topic, which is introduced with this article. As the first essay mentioned, “Facebook’s design—really, the design of public and semi-private virtual interaction spaces on the web—is starting to feel like it’s reached its past-due date.” While I think there is a future for social media, the act and necessity of code switching is a tiring one.

As more media show up, we’re finding different ways to interact on each one and access different groups through our social media channels–ideally, we’d only need to code switch if we app switched. Personally, I’ve found my Facebook interactions have transformed since I started using the service–I primarily interact with a few specific friends on their walls/timelines, engage more broadly with a few Facebook groups, and the content that I share most broadly (primarily links) still excludes some friends.

Code switching and social media also calls attention to the more limited self-presentations that we share widely. In times of scandal or trauma, however, these limited self-presentations can be exploitatively elevated as a reflection of a person’s true character. I wrote about this some time ago, after the Boston Marathon bombings and manhunt that followed.

Often manifesting itself as a method of assimilation, code switching can be tiring to the point of emotional labor. Anil Dash, a prominent technologist, goes so far as to recommend code switching as a way for people (like women) who are underrepresented in the technology field to succeed. As he acknowledges, “even if a candidate can navigate this formidable set of obstacles, they might well end up in a career that’s unfulfilling, marginalized within the companies they’d worked so hard to join. The simple reason why? These candidates speak a different language.” His solution sparked some controversy, as it seems to suggest that women and racial and ethnic minorities voluntarily perform this emotional labor to be successful in the tech industry, and he acknowledges that at the end of his post after receiving feedback on Twitter.

When women and minorities “speak a different language”, that means they often aren’t well-represented at tech industry conferences, among others. In an essay for the new publication Model View Culture, Alexis Finch discusses ways for organizers of conferences dominated by white men to amplify the voices of these underrepresented groups. Anil Dash spent the last year attempting to do just that, very consciously. He chose to only retweet women, and shared the results of his experiment. I took advantage of a tool mentioned in his essay, Twee-Q, to find out how well I do with my retweeting habits on Twitter, and the results weren’t nearly as good as I was hoping they’d be. I discovered I had a twee-q of 6, with 30 retweets of men and 18 retweets of women.

Something I really love about Twitter is that it makes everything seem more accessible. (It’s an illusion, because Twitter has 241 million active users (as of this month), a mere 3.4% of the global population.) However, it has allowed me to tap into the thoughts, streams, and articles that other people are reading. I feel like I can get to know the people that write the articles that I read all day, and sometimes you can converse with them. I’m lucky enough to be interested in a topic that has a disproportionate amount of writers and academics on Twitter. For me, Twitter lowers the interaction boundary implicitly set by other forms of communication media–Facebook is typically reserved for friends, email is often overcrowded, professional, and feels like you’re flinging a missive into the abyss. Twitter, on the other hand, you can mention that you’re looking forward to reading someone’s book and get a chapter recommendation, or that you loved a talk they gave and they just might reply, or sometimes you’ll post an article and a quote, and the author will thank you.

I suppose it’s one of those things that if you put a lot into it, you may get a lot out of it. Of course, your ambition for success might be far greater than the skills that you possess. One of the traps that I fall into often (especially following the people on Twitter that I do) is comparing my blog posts against the ideas of 30 year old researchers with PhDs, or people with even more experience in the field. However, as Ira Glass reminds us, almost everyone struggles with this gap between skills and ambition. Someone named Daniel created a short film with this key message, and it really is great.

As we become further entrenched in using tools owned by others (which we don’t pay to use), it’s important to assess ownership, and the growing blur between what is private and what is public. As Anil Dash discusses in his lecture “The Web We Lost” it wasn’t always that way, nor does it need to be in the future. If you prefer reading, he’s also published essays the cover the same topics, beginning with a discussion of public space that is provided by private entities, most infamously Zucotti Park, “home” of the Occupy Wall Street movement. He goes on to discuss how the web used to be--the web we lost–and how it can be rebuilt. It’s a long talk, but I really do recommend it as he’s actually good at speaking and the lecture is very well-structured.

As an expansion on his discussion of privately owned public space, San Francisco is now infamous for the shuttle buses that many Silicon Valley-based companies provide to their employees. As this comic at The Nib points out, this system has a lot of benefits, but also leads to some questions, such as–is building ones own private network really better than investing in an existing public one? Wired goes further, noting that “In the years to come, it will be vital for people who care about cities to not only understand them — but communicate about them in ways we can all understand.”

Further on the topic of technology and ownership, Apple’s released a new operating system last October that it called Mavericks. However, as Stewart Sinclair discusses, “The very nature of Mavericks — open, wild, unpredictable — is ostensibly in direct opposition with the technological environment Apple cultivates in its operating systems. Techies call it the walled garden; to stick with surf metaphors, we might rather term it Apple’s private beach.” On the topic of private beaches, some private areas may not be as private as you might think.

Technology can also upend ownership and privacy, as two academics demonstrated when they combined publicly available satellite imagery with Google Street View to map all the swimming pools in an area of Los Angeles. As described by Joseph Lee, one of the researchers, ““It’s an art and mapmaking project that tries to highlight the growing issue of data privacy in our increasingly digitized and inner-connected world."” Privacy by obscurity is no longer possible.

As always, if you have any suggestions, feedback, or reactions, leave me a comment! This week’s newsletter was pretty tech-heavy (and pretty Anil Dash-heavy) but thank you for reading.