Take Good Notes

Software developers have to keep track of a lot of disparate things. When debugging a knotty problem, for example, the solution process might involve references to issues, source control branches, downloaded test data, snippets from Slack or email conversations, stackoverflow questions, API documentation, etc. If you can pull all those items together into a single, searchable place, you will be much more effective. For a long time I didn't have a system like that in place, and then one day, I did. This is how I got there.

I read David Allen's book Getting Things Done in 2007 or so, and it made a big impression on me. One of his big tenets is that you have an "inbox" of things you have to deal with, and periodically and in various ways you work your way through the inbox, either taking care of things right away (make a phone call, send an email, pay a bill, etc) schedule something for later, delete it (i.e. throw it in the trash), file it as reference material, that sort of thing.

I followed his methodology for a while, and it works well if you can stick to it, but one thing that hung me up at the time was the "file for reference" bit. By this time, most information was digital, not on paper, and filing it meant... what, exactly? Perhaps wisely, David Allen didn't give detailed advice on implementing his system using tools that would quickly make the book outdated, I could, of course, just save items as files to the hard drive, but how to organize them? How to find them later? How to reference one item from another? What if I wasn't on that particular computer when I needed to refer to them? I saw coworkers use various digital Post-It note programs at the time, but these seemed to suffer from some of the same problems.

One attempt was to create a personal Wiki. This was a decent solution for notes and information that I typed myself, or could paste from another source, but was far from perfect. There was still the wiki markup to deal with, and while it wasn't that complicated, it was still a bit of cognitive friction to just getting my thoughts down as rapidly as possible. If you wanted to add an image, it was a cumbersome process of uploading it, then referring to it.

Then there were paper notebooks. For years I had used composition books as "field notebooks" at work, logging thoughts on software designs, bug writeups, sketching architectural diagrams, etc. This worked reasonably well, except I like things to be quickly searchable. Oh, and writing longhand for me is slower than typing and the results are not particularly legible. (You may have heard about studies that found that hand-writing notes leads to better retention of the information. This may very well be true, but usually I'm not writing to retain or memorize.)

This is where things stood a few years later when a coworker gave her notice and bequeathed to me a collection of notes from some projects I was inheriting. She had exported the notes from Evernote, and the easiest way to access them was to create an Evernote account and import them. I had heard of Evernote and its brethren (OneNote, etc) but had not taken the plunge on a modern note-taking app. I created a free account, and was off to the races. I quickly started using it to take notes from work meetings and calls. I found it was very good for "software development field notes", particularly when troubleshooting problems. You could paste in exception text and log messages, and document the things you tried and what the results were. Then I started adding notes for feature ideas, or detailing a difficult setup procedure. 

I soon started using Evernote for non-work things as well. It's great for managing small- to medium-sized projects like "Buy a car" or "House electrical work".  Pretty soon it reached the point where it just contained the details of my life. Evernote's icon is an elephant, and that's apt - it holds all the details that my own memory certainly doesn't hold very well anymore (if it ever did).

What Evernote Gets Right

When you create a new note, you can just start typing. You don't need to worry about formatting, unless you want to. If the thoughts are pouring out of your brain, or you're taking meeting notes in real time, don't worry about typos, formatting, and capitalization - go back and fix them later. Or don't bother - it's your personal system, so do what works for you. Abbreviate long words you're using a lot, then go back and replace them later (or keep the abbreviations). For example, if you're writing a lot about microservices, just write "ms".

 

The note editor has the usual "light formatting" options such as bold, italics, bulleted and numbered lists, links, images, blockquotes, etc.

I wish there was a Markdown input mode for technical content. There is a code block formatting option, which on the OSX version you can trigger with the Markdown-like triple-backslash

(For some reason this is not available on the Windows version - you have to choose Style > Code Block from the right-click or Format menu, or use the keyboard shortcut Ctrl-Shift-L.)

It's very easy to paste an image into a note from the clipboard, which is especially useful for screenshots. It will also perform optical character recognition (OCR) on images, scans, and PDFs, which is mostly used when searching.

Your data is continually saved in the cloud, and synched to all of your devices automatically. It's available on all of your mobile devices, of course, which is great when you want to jot down a quick gift idea when you're out and about, for example.

Organizing Your Notes

 

You can organize notes however you like. You can set up separate notebooks for different realms (work, home, volunteering, etc) or use tags, or both. I mostly use tags, as things often spill across different contexts. Tagging works as you expect - there is a many-to-many relationship between tags and notes. I don't go too crazy adding tags, as the full-text search functionality works pretty well. Most notes tend to have one or two tags. Doing a full-text search combined with filtering on a tag is particularly effective.

I also keep the number of notebooks to a minimum - I have one primary notebook into which I put both current work and non-work notes. I tag anything work-related as "work", along with more project-specific tags. On those occasions when I change jobs, I archive the notes from the old job by moving then to their own notebook. 

Another really useful tactic is to make your own wiki-style links between related notes. This is great for adding "see also" links, and for breaking up notes that are getting too long into several.

Checking The Box and Searching

You can also include checkboxes in your notes, which gives you the ability to make to-do lists and the like:

Naturally, there's a keyboard shortcut for that - Ctrl-Shift-C on Windows and Cmd-Shift-T on Mac. You can also just type a pair of square brackets (like how you would show a checkbox in ASCII) and it will automatically turn it into a checkbox for you.

Particularly when you're working on a greenfield project, you tend to have lots of to-do items that you need to come back to. Sometimes TODO comments in the code or the readme are the best way to do this, but sometimes it's easier to dump them out into the note you're in the middle of, especially if they're not related to code changes.

 

You can set up customized searches within Evernote. A couple I find particularly useful are:

• Work To Do - all notes tagged work that have at least one unchecked checkbox
• Non-Work To Do - all notes NOT tagged work that have at least one unchecked checkbox 
• Not Work - mostly used to find recent notes that I've forgotten to tag as "work"

I spend a good part of my workday in Evernote, so I put a little extra effort in making sure my future self can find things quickly. For example, if I take notes while troubleshooting a bug, I put the ticket number in the note title. Or, if there is a particular thing that is referred to in different ways, like by abbreviation or fully spelled-out, I make sure both are included in the note so searching on either form will locate it.

Outside of Work

I end up taking notes for a lot of non-work stuff, too, such as medical history, food diaries, car and house repairs, project ideas, book summaries, etc. It's especially good to keep track of gift ideas for friends and family (or yourself!), and lists of books to read or movie recommendations.

Clipping It

One nice feature of Evernote is its Web Clipper browser plugin, which allows you to save a snapshot of a web page as a note. This helps to solve the "too many browser tabs open" problem. When you find a page that has a lot of good reference information in it (in the GTD sense), you can clip it. It does a very good job of removing the menus and other fluff and keeping just the body of the page. Since it saves a copy of the content, you're protected against the page going away in the future or moving to a different location.

Useful Keyboard Shortcuts (Windows Edition)

If you're a developer who doesn't use a lot of keyboard shortcuts, you may have a vague sense that you should be using them more, or you've read that the "cool kids" (i.e. the really smart and productive developers) use them all the time and hardly ever touch the mouse. I'm here to tell you it's true - it does make you more productive.  A little effort to use more keyboard shortcuts and fewer mouse actions really pays off.  Just by speeding up common actions, you can better keep up with your own brain. Even if you like your mouse, if you're on a laptop without a mouse, you may find the keyboard shortcuts less annoying (and more accurate) than using the trackpad (or the little pointy-thingy in the middle of the keyboard).

 

Windows Key Shortcuts

I went years completely ignoring the humble Windows key, but it turns out it does a lot of useful stuff!

 

Windows key by itself

Tapping the Windows key brings up the start menu, and places the cursor right in the search/run box. This is a great way to launch apps or files, or do searches. It also has nice autocomplete functionality. Note that Ctrl-Esc does the same thing, but why use two keys when you can use one?

 

Windows key combinations

Win-X -This gives you a quick menu with a lot of useful system actions:

Win-Up - Maximizes the current window.

Win-Down - If current window is maximized, will change it to "normal" mode (unmaximized). If unmaximized, it will minimize the window. So you can go from maximized to minimized by doing Win-Down twice.

Win-Shift-Left, Win-Shift-Right - Moves the current window to the next monitor to the left/right. I'm not a big fan of three-key combinations (sometimes called "chords") but this one is an absolute godsend if you use multiple monitors.

Win-L - Lock the screen. Quicker than ctrl-alt-del, then choosing "lock screen". This is handy in a work environment when your company security policy requires you to lock your workstation when not at your desk, or when working in any other public place.

Win-M - Minimize all windows. Handy for "tidying up" when you have a lot of windows open all over the screen.

Win-Left, Win-Right - Tile the current window to the left half/right half of the screen. This is particularly useful when you've only got one monitor, but you need to refer to two apps at the same time. Also useful if you want to compare two versions of something side-by-side, but for some reason you can't use a diffing tool. Maybe you have two similar Word documents or something like that.

 

Alt Key Shortcuts

Alt key shortcuts also tend to do things at the operating system level, rather at the individual application level.

Alt-Tab - Switch between open applications/window. This is the big enchilada - it allows you to switch to a different window quickly. Doing a quick alt-tab and releasing will switch to the last window you used, which is very handy for switching back and forth between two windows. Holding down the Alt key, you can keep hitting Tab and it will cycle through all the open windows, or you can use the arrow keys to navigate more directly to the one you want.

If you have multiple screens, it can be difficult to find the mouse pointer sometimes (at least for me), and this slows me down if I want to switch to a new application by clicking on the app on the taskbar. Alt-Tab lets you go to the application you want to without needing to know where the mouse is - when you finish, you have focus in the application.

Alt-F4 - Closes the current window. This is very useful, but unfortunately I find this a bit of a "finger stretcher." Side note: why doesn't Alt-W close a window, analogous to how Ctrl-W (see below) closes a document within an application?

Alt-Spacebar - This displays the current window's System menu:

I don't use this often, but it's really useful if you're in the catch-22 where the top of the window is off the screen and you need to grab the top of the window in order to move it all onto the screen.

 

Ctrl Key Shortcuts

Control (Ctrl) key shortcuts usually work within an application, and fortunately there's a lot of standard behavior that works in most programs.

Ctrl-X, Ctrl-C, Ctrl-V - Cut, Copy, Paste. If you're not already using these, stop right now, learn them, and practice them so you'll never need to go to the Edit menu for these again.
Ctrl-Z - Undo. What I said about Cut, Copy and Paste applies here as well.
Ctrl-A - Select All in current document or page
Ctrl-S - Save current document/file etc.
Ctrl-O - Open File
Ctrl-N - New (File, Document, etc.)
Ctrl-F - Find, usually "Find in current document" or "Find in current page".
Ctrl-H - Very often this is Find and replace in current document (or current page).

Once you get those down, these would be the next ones to start fooling with:

Ctrl-W - Close current tab or document (within multiple tab or multiple document application). In Visual Studio this is Ctrl-F4 by default (although you can change it to Ctrl-W).
Ctrl-G - In an editor, this is usually "Go to line"; i.e. typing "Ctrl-G 47" will take you to line 47 of the current file.
Ctrl-I - In an editor with formatting capabilities, this will often toggle italics on/off for the selected text.
Ctrl-B - In an editor with formatting capabilities, this will often toggle bold on/off for the selected text.
Ctrl-Tab - In multiple-document or multiple-tab applications, often cycles through tabs or allows you select tabs or open documents similar to how Alt-Tab works.
Ctrl-T - in an application with multiple tabs, this is often "New Tab".
Ctrl-+, Ctrl-- - Increase or decrease font size. In particular, "Increase Font Size" is extremely useful for those of us who have reached, shall we say, a certain "maturity level".
Ctrl-K - In an editor or email client, this will often allow you to set a hyperlink for the selected text. If you have the URL in the clipboard, this then becomes a matter of selecting the text, then pressing Ctrl-K, Ctrl-V

 

The Rest - Navigation, etc.

If you start using more keyboard shortcuts, you may notice that "keyboard shortcut use begets more keyboard shortcut use." In other words, once you start keeping your hands on the keyboard more, you may find yourself looking for more ways to keep them there. This may happen in part because once you've been ignoring the mouse pointer for a few minutes, you may have trouble finding it again, particularly if you have multiple screens.

Aside: To help with this, you can change your mouse settings so tapping the Ctrl key will give you a visual indication of where the mouse pointer is. Here's what the setting looks like:

Here are some other keyboard actions to help you stay on the keyboard.

Tab - go to the next input field in a form. Shift-tab goes to the previous one.
Enter - click the button with the current focus
Esc - Cancel the current dialog or operation. You probably already knew about this one, but you may find yourself using it a lot more if you're more keyboard-oriented.
Arrow keys, PgUp, PgDown, Home, End - Likewise, use these to navigate through a document or page in a text editor, browser, email client, etc.
Ctrl-Left Arrow, Ctrl-Right Arrow - Navigate one word left or right.
Ctrl-Home - Go to the top of the document or page
Ctrl-End - Go to the bottom of the document or page

 

Learn a Few at a Time

Rather than trying to learn a lot of these at once, what has worked better for me is to focus on a few, getting them "under my fingers," then moving on to others. Little posted cheat-sheets like this are helpful:

 

Browser Shortcuts

Everybody spends a good bit of time in a web browser, so knowing a few shortcuts there can really pay off.

Ctrl-T - Open a new tab
Ctrl-W - Close current tab
Ctrl-Tab - Go to the tab to the right of the current tab (this is sometimes configurable depending on the browser and plugins. For example, it may take you to the last tab used)
Shift-Ctrl-Tab - Go to the tab to the left of the current one
Ctrl-L - Go to the location bar, and select the current URL (so if you just start typing a new URL or search phrase, it will replace what's there)
F5 - Reload the current page
Backspace - Go back to the previous page (or previous point in history). In other words it works like the Back button.
F12 - Open Developer Tools

 

Application-Specific Shortcuts

If you've come this far, you're ready to really optimize. Figure out what tools you use the most, and then work to get familiar with those tools. You probably want to get good with the keyboard shortcuts in your email client, whatever it is. If you're in finance and you spend all day in Excel, optimize the crap out of Excel. Part of being a professional is becoming an expert at the tools you use in your profession every day.

Logging and Error-Handling Guidelines

I'm pretty sure I sound kind of logging-obsessed to my fellow developers sometimes. I'm sure I sound like a broken record sometimes when I ask to see the log when troubleshooting an issue, but when logging is done right it provides a clear path toward getting to the root cause of a problem. To my mind, this is about 1000 percent better than the alternative, which mostly consists of guessing and randomly trying things, followed by hiding from management when they come around demanding a status update.

So, logging is extremely valuable- at least good logging is. But what makes for good logging? I'm tempted to say "I know it when I see it", but in fact over the years I have managed to codify some guidelines about logging and error handling, which I'll lay out here.

We use log4net as our standard logging package for .NET projects. It's been around forever, and I know there are sexier options out there, but the guidelines below kind of assume log4net. If you use a different logging framework, I'm pretty sure most of the recommendations will still apply.

In a normal production environment, the logging level in configuration should be set to INFO. In log4net XML configuration, it would look something like this:

  <log4net>
    <appender name="TheAppender" etc.. />
    <root>
      <level value="INFO" />
      <appender-ref ref="TheAppender" />
    </root>
  </log4net>

When you are testing or troubleshooting, the level can be changed to DEBUG to gather more information about the system. That means that as you write log statements, you'll need to think about whether you'll want to see them generated as part of normal behavior, or only in debugging/troubleshooting situations.

Each class should have its own Logger instance. We declare it like this:

private static readonly log4net.ILog Log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType.Name);

This allows the logger to automatically pick up the name of the class, and including the fully-qualified class name on everything allows it to be copied "as is" into a new class in one action without needing to add any "using" directives.

When coding, try to anticipate what types of log messages that would be helpful to a future troubleshooter. This is a bit of a balance, because you want to avoid "spamming" the log with low-value information.

Log every request coming in at the highest code level with a level of INFO. This means controller methods, Main(), etc. This gives you a permanent record of all the requests that you handle.

When logging unexpected error conditions or exceptions you can't recover from, do so with a logging level of ERROR. This may sound obvious, but if you're consistent about it, you can quickly search through any log for the problem that Support is probably asking you about.

Exceptions that are logged should be done so with the full details of the exception, including message, stack trace, inner exception (if any), etc. Log4net makes this easy. Having the full stack trace in the log is enormously valuable for troubleshooting.

Log every condition that can be handled but may be a sign of trouble with a logging level of WARN. These often provide the "smoking gun" for more inscrutable errors.

When you get bug reports around unexpected edge cases, log the condition when you're fixing the issue. These are often WARN statements. 

Log every database read and write (in Entity Framework, etc.) with a logging level of DEBUG. These are almost always too frequent to log as INFO (and can leak sensitive information) but are enormously helpful for troubleshooting.

Log other interactions with the outside world (file interaction, communication with external web services, etc.) with a logging level of INFO, unless they're so common they really spam the log, in which case go with DEBUG. The same reasoning as for database access - out-of-process calls of any type are a frequent source of failure.

Log any other helpful data, as needed, with a logging level of DEBUG. These types of log statements tend to be up to the discretion of the developer, and often they may be removed once a section of code is stable.

Errors that are not caught anywhere in the code should be caught and logged by the framework if possible. ELMAH is one tool that does this for ASP.NET web applications. If not possible, the entry point method (controller method, Main()) should have a general catch exception that logs the error.

If exceptions are caught, they should be re-thrown or completely handled. Under no circumstances should they leave the data in a bad state.

Try/catch blocks should be relatively rare and exist to handle specific error conditions. I've seen a lot of code where developers put a try/catch block in almost every method. This just adds cruft and overhead without adding any value. I think in most cases developers who do this are falling into a "cargo cult" mentality where they think they must add try/catch blocks everywhere in order to "do error handling right." They may be half-right - they should think about error handling in every method, but in my experience most of the time the best thing to do is simply let the exception be thrown and handled up the call stack.

Do not return status codes to indicate whether there was an error in a method. If there was an error, throw an exception from the method. This simplifies the calling code significantly as it does not have to check the return values or status codes of method calls. Older programmers who started out their careers using C or other languages without exception support are probably the most guilty of this.

Sometimes it makes sense to catch exceptions at a lower level in order to log specific details about the exception at that level. If this is done, be sure to re-throw the exception.
When re-throwing an exception, it is usually better to use “throw” by itself, rather than throwing the caught exception. Here is a quick example:

string base64 = …
byte[] bytes;
try
{
bytes = Convert.FromBase64String(base64);
}
catch (FormatException ex)
{
// Log the bad data for help in troubleshooting the problem
Log.ErrorFormat("Bad base64 string: '{0}'", base64);
throw; // this preserves the full stack trace
// do not do this as it resets the stack trace:
// throw ex;
}

In the code above, the only reason there is a try/catch block is so we can log the bad base64 string in the case that it cannot be converted to a byte array. By calling "throw" rather than "throw ex", the stack trace will appear upstream as if it were not caught at all at this level.

Avoid NullReferenceExceptions as they typically do not contain adequate information about what went wrong. Validating method arguments and throwing ArumentNullException, ArgumentOutOfRangeException, etc. is the best practice to avoid this.

Along the same lines, be sure to validate the result of data queries, including Entity Framework or LINQ queries, and throw an appropriate exception. An exception of this type ("Contact #123 was not found in database") is much more helpful than a NullReferenceException occurring many method calls later.

Client applications need to be able to handle various failed requests to the server (server not responding, 404 errors, 500 errors, etc.). If this is not done, it often leaves the client application "hanging" in an unresponsive state.

Taking a step back, you'll want to think about the process around all this. When a bug or problem is discovered, part of getting it fixed or resolved should be a discussion of what additional logging or instrumentation would have helped to diagnose the issue. Since you generally can't (or shouldn't) do source-level debugging on a production system, you have to rely on production logs, which should help you reproduce the issue in a development environment.

My Queue-Based Life

"In the future, there will be so much going on that no one will be able to keep track of it."

--David Byrne, "In the Future", from The Knee Plays

I love Instapaper. It helped solve the "20 browser tabs open" problem where you click on something interesting to read - but it turns out to be kinda long, and you don't have time to read it right now. In fact, you don't really want to read it on your work computer at all; you'd rather pull it up on your tablet in your living room with a fire going in the fireplace and a glass of Dogfish Head 60-Minute IPA at your side. Consequently, I tend to have a fairly long list of interesting articles queued up to read at any given time, lovely articles from The Atlantic and Slate and The New Yorker and Vanity Fair and Wired and Rolling Stone and qz.com and the like.

However, the Instapaper articles compete with digital issues of Newsweek, which are full of articles about NEWS which come out EVERY SINGLE WEEK. I'm usually about 6-8 issues behind, which means that I can sometimes skip articles because the Big Question they're asking has been answered, or is no longer relevant at all.

Newsweek and Instapaper, of course, compete for my reading time with books, still mostly paper but some digital. Authors have this annoying habit of writing books on fascinating subjects that I can't wait to dive into, and so those titles are added to the list I keep on a Google Drive spreadsheet, which is also more or less duplicated in a Goodreads account. Many of these books have been purchased and sit there, waiting to be read as soon as I can get around to them.

If I don't feel like reading, no problem - TiVo dutifully records all kinds of interesting TV programs, hours and hours and hours worth, more than my wife and I could ever hope to watch. Lately, an "AFI's List of 100 Greatest Movies" smart search thingy has intersected with an enhanced cable subscription that gives us Turner Classic Movies, and the TiVo lit up like a pinball machine, recording "Casablanca" and "Citizen Kane" and "The African Queen" and "Bringing Up Baby" and "Platoon" and "The Last Picture Show" and so on and so on.

If we're not in the mood for something on TiVo, Netflix has us covered - there are maybe 50 movies and a few TV series in the queue there. Who knows if we'll ever get around to watching them? Actually, I know - we'll never get around to watching them all. And that's not including the dozens of TV shows or movies that my friends tell me I MUST watch - how could I be missing out on Game of Thrones or Fargo or The Walking Dead or Adventure Time or Girls or Black-ish or Doctor Who or The Knick, etc, etc, etc.

Meanwhile, when I'm exercising or driving or doing mundane household chores or just lying sick in bed, there are the podcasts. I've discovered I'm something of a podcast junkie. It is so, so easy to subscribe to a new one, promising hours and hours of interesting listening - techie podcasts like Hanselminutes and .NET Rocks; wonky podcasts like Planet Money and Freakonomics and The Commonwealth Club of California; the Slate Political and Culture gabfests; Marc Maron's WTF interviews, and just damned interesting podcasts like Radiolab and 99% Invisible. I'm months behind on some of those.

This may be the ultimate First World Problem. There is so much quality content being produced, by so many amazingly talented writers/thinkers/researchers/producers/directors/actors/singers/playwrights/I/could/go/on/and/on that I occasionally stress out that I'm not consuming it fast enough. Each separate stream of content threatens to become its own to-do list to be worked through as quickly and efficiently as possible, like Lucy and Ethel wrapping candies at the conveyor belt.

I have, sometimes, learned to let go, not worry about all of it piling up, and just cherry-pick the best stuff .This doesn't solve the larger problem, however, which is that consuming all this stuff - as great as it can be - is not the same as creating something or doing something. Yes, I am an info-junkie, but i have to remember to put that information to good use in the service of action, not merely learning for learning's sake, as attractive as that can be.

Creating a Shuffled Classical Music Playlist

I have a reasonably large classical music collection, and sometimes I'm in the mood to put classical music on, but I don't want to choose exactly what. Yes, I could find some classical music radio station, either over the air or on the internet, but having grown up in an age before universal streaming, sometimes I just want to listen to my music. I'm generally a big fan of randomly shuffled music, and I like Smart Playlists in iTunes. Using smart playlists, I can set up metadata-based playlists like "play a random selection of all rock or blues tracks rated 4 stars or higher than have not been played in the last 3 months."

Classical music is different than rock, pop, blues, jazz, country or most other forms of music. Besides the obvious things, you're often more interested in the composer rather than the performer. For example, I know I like Beethoven better than Vivaldi, but I wouldn't say that I like the Boston Symphony better than the New York Philharmonic. (This isn't 100% true, particularly for solo artists - I might have particular pianists, violinists, etc. that I really like.)

You'll notice another difference with classical music if you try to listen to it by shuffling tracks. You might get the second movement of a Mendelssohn symphony, followed by the third movement of a string quartet, followed by the first movement of a piano concerto, followed by the William Tell Overture. This is not quite what I'm looking for - I'd rather have it play the entire symphony, followed by the entire string quartet, followed by the whole piano concerto, followed by the William Tell Overture.

I could shuffle by album, but that's not quite "shuffle-y" enough. Plus, occasionally I have a piece that spans two discs. I have a nice recording of Tchaikovsky's Symphonies 4, 5, and 6 on two discs, but the poor Fifth Symphony gets split across the discs.

In other words, I don't want to shuffle by track but by work. A classical work is the typical unit of composition and performance - Beethoven wrote the 5th symphony as a cohesive whole, and usually all four movements are performed together and in succession.

Fortunately, iTunes supports classical composers and works, sort of. If you right-click on a track and choose "Get Info", you'll see something like this:

There is a specific field for Composer, but what about Work? Well, it turns out that iTunes treats the Grouping field as the work, at least for tracks in the Classical genre. You used to be able to see this in the Classical Music smart playlist in older version of iTunes, but apparently they've changed things around in more recent versions so it doesn't say anything about the work there anymore.

To aid in the effort of adding composer and grouping information, I made smart playlists called "Classical with No Composer" and "Classical with No Work Grouping", which help to identify where I need to add metadata:

It's relatively easy to add composer information to everything, but quite a bit more work to add Work groupings if you have a large classical music collection. It turns out that having the Work entered for everything is nice, but not critically important, as I'll get to in a minute.

What I want, then, is to get all the tracks in the Classical genre that have a composer, group them by work, then shuffle the works, while preserving the track order within the work. Building on some earlier work coding against the iTunes API, this ultimately evolved into a high-level LINQ expression that's fairly expressive:

                 List<IITFileOrCDTrack > tracks =

                    iTunes.Library.FileTracks

                        .InGenre( "Classical")

                        .WithComposer()

                        .WithFile()

                        .ShuffleByWork()

                        .ToList();

  

This expression uses several extension methods on IEnumerable<IITFileOrCDTrack>:

         public static IEnumerable< IITFileOrCDTrack> InGenre(this IEnumerable<IITFileOrCDTrack > trackCollection, string genreName)

        {

            return from IITFileOrCDTrack track in trackCollection

                where track.Genre == genreName

                select track;

        }

        public static IEnumerable< IITFileOrCDTrack> WithComposer(this IEnumerable<IITFileOrCDTrack > trackCollection)

        {

            return from IITFileOrCDTrack track in trackCollection

                   where !string .IsNullOrWhiteSpace(track.Composer)

                   select track;

        }

        public static IEnumerable< IITFileOrCDTrack> WithFile(this IEnumerable<IITFileOrCDTrack > trackCollection)

        {

            return from IITFileOrCDTrack track in trackCollection

                   where File .Exists(track.Location)

                   select track;

        }

  

These are pretty straightforward, but obviously the ShuffleByWork method is the hard part. But first, we need a simple ClassicalWork class:

    public class ClassicalWork

    {

        public string Composer { get; private set ; }

        public string Album { get; private set ; }

        public string Name { get; private set ; }

    }

  

This class also has a constructor, plus the System.Object overrides Equals(), GetHashCode(), and ToString() methods, but nothing spectacular here.

The ShuffleByWork() method starts out by grouping all the tracks into ClassicalWork objects:

             var workGroups = from filetrack in filetracks

                             group filetrack by new ClassicalWork(filetrack.Composer, filetrack.Album, Classical.WorkName(filetrack.Grouping, filetrack.Name));

  

The Classical.WorkName() static method constructs the best possible name for the work. If we have a Grouping for the particular track, well, we know that's the work name, so we go with that. If not, we look at the track name. Overtures are normally only one track, and are good to intersperse between longer works, so if the track name contains "overture" return it as the work name. Otherwise, we just an empty string, which means that this track isn't really part of a work (but we'll still include it in the shuffled playlist).

Grouping tracks in this manner is really what defines what we consider to be a work. All tracks on a particular album that have the same composer and work name (as defined above) are considered to be the same work as far as our shuffling goes. This means that all tracks on an album that have the same composer but don't have the same Grouping will still get grouped together into one pseudo-work. In practice, this is fine - if I have several tracks on an album by Sibelius, say, but don't know anything else about whether they fit together, it's perfectly OK to group them together.

So now workGroups is a list of all the works we're dealing with, each with a list of the tracks in the work. Next, we just shuffle all the work groups:

             var shuffledWorkGroups = workGroups.Shuffle();

  

The Shuffle() extension method is perfectly generic (in both senses of the word) and could be used to shuffle any collection - a deck of cards, etc:

        public static IEnumerable<T> Shuffle<T>( this IEnumerable <T> enumerable)

        {

            var random = new Random();

            var shuffled = from item in enumerable

                           orderby random.Next()

                           select item;

            return shuffled;

        }

  

The last thing to do in the ShuffleByWork() method is to iterate through each item in our shuffled work groupings, and return the tracks in each grouping ordered by the original track number:

             foreach (var workGroup in shuffledWorkGroups)

            {

                // Work name is workGroup.Key;

                var tracks = workGroup.OrderBy(t => t.TrackNumber);

                 foreach (IITFileOrCDTrack track in tracks)

                {

                    yield return track;

                }

            }

  

Next, we'll create a playlist, deleting an existing playlist if necessary:

                IITUserPlaylist playlist = iTunes.FindPlaylistByName(playlistName);

                if (playlist != null )

                {

                    playlist.Delete();

                }

                playlist = iTunes.CreatePlaylist(playlistName);

                foreach (var track in tracks)

                {

                    playlist.AddTrack(track);

                }

Once the program runs, the playlist obligingly shows up in iTunes:

Finally, we'll create an M3U playlist, which is a very simple text file and will allow non-iTunes apps to access the data. 

We'd like to shuffle this on a regular basis, so I need to create a little batch file that can be run as a scheduled task every night. The essence of that is this line:

ClassicalPlaylist.exe "Classical Shuffle" "%USERPROFILE%\Music\My Playlists\Classical Shuffle.m3u"

With that done, I can point all the various music devices and services in my home (Sonos, Plex, etc.) to that .m3u file,

The project and source code is up on Github if you want to take a look.

Support and Troubleshooting Techniques for Software Developers

Developers at the company where I work, including myself, do a regular rotation as 2nd or 3rd level product support. Basically, whenever our regular customer support folks have support questions they can't resolve on their own, they reach out to the on-call developer. As you might expect, this is not our favorite part of the job, but sometimes it's rewarding. To it a little more bearable, sometimes I think of it as detective work. Think of a police procedural TV program like "Law and Order", where the detectives going around interviewing eyewitnesses to the original crime, which produces more leads, etc.

I'm assuming at this point that a support person has come to you with a customer problem that they can't solve. At this point, your job is to go where support people generally can't - into the code.

Root Cause Analysis

Here's my main point: Work Backwards. Yes, I said this in the last post as well, and I'll repeat the rest of it: Start at the point where the error first shows up, figure out what condition(s) cause that to occur, then figure out what causes that, etc. Methodically work your way up the chain of causality. Don't attempt a fix without any analysis, unless you have a really good reason to believe it will work in this situation - in which case, you're not troubleshooting, you're just applying a remedy to a known issue. If support didn't know about the issue, be sure they add it to their knowledge base.

Start with some Basic Information 

I probably sound like a broken record to our support team, because I'm always asking for two things right off the bat:

1. What version of the software is the customer running? If you have a single, web-based application that everyone in the world hits, this won't be a concern.

2. Can you get me the logs?

I need to know what version they're on so I know which version of the code to pull to start the causality investigation. I want the logs because the logs usually tell me a lot more about the error than the user-facing error message does (and sometimes there's no user-facing error message at all, just a "something didn't save correctly" or "so-and-so didn't respond." If it's an exception thrown, the logs will give me the full exception stack trace, which is invaluable. Again, this is the way we tie the error to the code.

If your software does not log errors when they occur, stop right now and make it a high priority. Do not release your next version without it. If you encounter resistance on this, start looking for another job, because I can't think of many things more frustrating than having to support software that doesn't have a way to tell you when errors occur, and what the nature of those errors is.

Some other things to ask the customer (or have Support ask the customer):

• If something had been working and suddenly started having problems, what has recently changed in the environment? New hardware, new software, version upgrades, etc.?
• Is the problem isolated to one user? If so, start investigating what is different about that user. Insufficient permissions?
• Is the problem isolated to a single workstation? If so, same thing - what's different about that workstation?
• Is the problem isolated to a single piece of data? Perhaps some data got corrupted.

Be Methodical

Don't be in a too much of a hurry - if you follow the steps, you'll almost always eventually come to the right answer. Apply logical reasoning: as a software developer, that's a big part of what you get paid for.

For example, if the user is seeing a particular error message, I need to know exactly what the error message says - preferably a screenshot or a text copy of the message. As a developer, your job is to tie the error back to the code. Sometimes you'll know exactly where in the code to look, but often, especially if your codebase is large, you'll need to search for the error text in the code.

Static Analysis

Once you've found the line of code that produces the error message, start doing some static analysis. Use your IDE's "find usages" or "find references to" functionality (or in the worst case, the straight "find" functionality) and start going up the call stack. The possible call stacks may start fanning out, but if you cross-reference against log messages, plus what you know about the configuration of the system, you'll usually have a good idea of the code path. You may need to go back to the support team or the client to get more information.

In modern AJAX web applications, the error may first be thrown on the server, but the call stack will eventually lead to a web service call, and from there into client-side JavaScript. The same rules apply, although the functional, untyped world of JavaScript can make static analysis a bit more difficult.

If you can't find the error or exception message in your code, it's probably coming from a third-party library you're depending on. Next stop: Google (or your favorite search engine) or maybe stackoverflow. Once you find the library function call involved, look for where it's called in your code, then follow the call stack up as before.

OK, That Didn't Work...

If nothing has panned out so far, here are some other things to try. These are roughly in order of easiest-to-hardest.

• Is this a known issue? Check your bug database, release notes, knowledge base, etc. Yes, support should have done this, but you probably have access to more resources (the bug database, for example). Also, with your perspective as a developer, you may search the knowledge base in a different way and come up with different (sometimes better) results.
• If the error is repeatable but you might be able to bump up the logging level (turn on Debug logging, for example), then reproduce the error and get the new, enhanced log. The debug messages may tell you enough about what's going on to determine the problem.
• Sometimes you'll need to look at the change history of code, to find out when something was broken (or was fixed). If your changesets are tied to tickets, ether enhancements or bugs, it will further shed light on why a particular change was made.
• See if your testing team can repeat the error. If so, they'll probably enter it as a defect.
• See if you can repeat the error in your development environment. If so, you can probably capture the problem in the debugger, which will give you much richer information that the static analysis approach detailed above.

What To Do If You Get Stuck

• Brainstorm the problem with your colleagues. They may remember a crucial piece of information you don't have, or come up with an approach you haven't thought of.
• Relax your assumptions. If all assumptions were valid, the application would be working perfectly, right?
• Some problems are really hard to diagnose because there are multiple problems interacting. Sometimes this is because there's a bug in the error-handling code
• Remember that everyting is obeying physical laws, and there is a rational cause and effect going on. Do not treat the system as though it is magical.
• That said, occasionally problems happen that you just can't explain. Hopefully this is very rare.

Workarounds

If it's a problem in production, you may actually have two separate tasks: getting it working ASAP. This may involve manually fixing some data in a database, or it might require a workaround or temporary expedient of some kind. If the problem is due to a software bug, making sure a bug report is entered so it gets triaged, prioritized, and (hopefully) fixed at some point.

Make It Easier to Troubleshoot Next Time

If the logging or other visibility into the code is inadequate, making the issue difficult to troubleshoot, the bug report may be "the so-and-so module has insufficient logging for troubleshooting and support," or better yet, something more specific, e.g. "The Flooper module doesn't log the full path of the file when it gets a 'cannot open file' error."

One thing I've learned about logging over the years is you definitely want to be able to log every significant interaction with the external environment: file opens, database interactions, web service calls, etc. You can't control these external things, but you can control exactly how you call them. In the worst case, these log messages could serve as a helpful bug report to the developers of the external module, but in most cases, you'll find the problem is with your own code.