This piece describes the original approach to the HDF Compass application. Compass has since been acquired by the HDF Group, and released as open-source software, so it's likely the current version will be quite different by the time you read this.
HDF5 is a file format and set of libraries for storing big numerical datasets. It's used widely in the scientific community, and increasingly in other fields with high data demands, including finance and statistics. While at UCLA, I started the h5py project, which allows simple access to HDF5 data from Python.
HDF Compass is a new, minimal viewer application for HDF5 and related formats, written in Python. It began as a commercial project for my consulting company, Heliosphere Research. This was the first full GUI application I ever worked on, and it was a highly instructive process.
Early in 2014, I had done some training for the HDF Group in using the h5py library. Shortly after, I had dinner with Ted Habermann and a couple other people from HDF, and the subject of HDFView came up.
For those who aren't immersed in the HDF ecosystem, HDFView is the standard viewer for HDF5 and HDF4 files. It's written in Java, so it runs on Mac, Windows and Linux, and supports a wide variety of features like HDF-EOS constructs and and dimension scales. Here's a screenshot:
Cross-platform compatibility is great, and HDFView leverages the existing HDF-Java library maintained by the HDF Group. But like a lot of cross-platform apps, and Java apps in particular, the user experience is not great. On the Mac, some behavior is wildly inappropriate... menu bars are attached to the frames instead of at the top of the screen, weird java widgets are used everywhere, and it doesn't work right with things like drag & drop of files onto the app icon or the Dock.
I use HDFView a lot in my research work, as a fast way to look inside HDF5 files and do some simple visualization. In a fit of enthusiasm, I had earlier started on a new viewer program written in Python, with a view to launching it on the Mac App Store (yes, really). Following the dinner with Ted and company I decided to dust off the code and ended up making a new, thoroughly minimal viewer program.
I had done a lot of Python programming through the h5py project, and in my daily work as a research scientist. But I'd never written a complete GUI app. Before getting into the code, I thought it would be a good idea to get the general concept figured out.
The first step was to take a look at other tools in the HDF universe. There's a screenshot of HDFView earlier in the article. Here's a screenshot of another HDF5 browser app, written in Python:
This is ViTables, a viewer specifically designed for looking at PyTables files, although it does an excellent job with plain HDF5 files as well. There are lots of others, including tools like Panoply, HDF Explorer and even the built-in browser in programs like IDL.
Looking at the variety of tools, a few common features stood out. Not all tools had these aspects, but most had some:
A tree control on the left-hand side. HDF5 files are hierarchical, so they map well to a tree metaphor. Each of the viewers above displays a hierarchy of objects in the file. You interact with objects by clicking or right-clicking objects in the tree, and browse the file by "expanding" nodes in the tree.
MDI interface. A bit of an older approach to UI design, this just means the objects appear as windows within the main application. The advantage is that you don't get lots of top-level windows showing up in your taskbar or in the Dock. The disadvantage is that things get cluttered pretty quickly.
Tiny Unlabeled Button Syndrome. This is somewhat a matter of taste, but many scientific tools suffer from a roaring case of TUBS. Toolbar buttons are a favorite of power users, and can serve as a useful shortcut for things like opening files or performing common operations like cut and paste on objects. But my goodness do they make the UI intimidating.
Editing support. I find this one really impressive. Many applications above supports interactive editing of files. That means changing data values, moving objects around in the file, creating new objects and attributes... Some even support Undo/Redo!
Let's start with the earliest build I could find of the app. The GUI is mostly mocked-up in wxPython; for example, buttons like "Cut" and "Paste" don't do anything:
I'm almost embarrassed to show this. At the time I thought the Ribbon was the best thing since sliced bread. You'll notice that as with HDFView, there's something like a URL bar, in this case showing the address of the current object in the file. Presumably so people could type in, character-by-character, the exact path to the object they wanted.
And the more I thought about it, the more became clear that there were already lots of full-featured tools available for working the HDF5 files, and there was simply no room for another interactive editing app with lots of buttons and a tree control on the left-hand side.
But the project, at this point a half-baked commercial app of questionable marketability, ground to a halt.
So, when restarting the project in 2014 I decided to take an opposite approach, and see where a minimal design would lead. I threw out the current version of the app and drew up a list of criteria, in rough order of importance:
Focus on viewing, and not editing. Everybody likes to quote Steve Jobs on the value of focus, so I decided to put my money where my mouth was and say "no" to editing support. This was a tough one, and the first decision I made. Focusing on making a kickass viewer and visualization program would free up resources to make those features as good as humanly possible, by making the code much, much simpler and reducing the testing time.
Adhere to modern platform norms. Native widgets. No more MDI. On Mac, the menu bar should be in the right place, at the top of the screen. When you drop a file onto the app icon, it should open. And goodbye to the Tiny Unlabeled Buttons.
Stand-alone. For example, the stock HDFView app is now distributed as part of the HDF-Java product. Hands up everyone who wants to install HDF-Java. The app should come bundled in the correct format for the platform... on Windows, as a .exe file or installer; on Mac, as a .app bundle.
File-browser interface style. Part of me wanted to see if people would lose their minds if they saw an HDF5 viewer without a left-side tree control. But I also really like the Finder on Mac, and wanted to see where a browser-style interface might lead.
Decent basic visualization. People should be able to view their data right in the app. The plots should be professional-quality, to the point where, although simple, they could be shared with colleagues without needing to fire up Python or MATLAB.
I decided to ditch every piece of the interface that wasn't absolutely essential. Here's the earliest screenshot I could find, just after I finished ripping everything out. You'll notice this is showing the filesystem, since the HDF5 bits weren't working yet:
Ignore the vomit-green buttons for a moment. At this point the basics of the new design were nailed down:
Navigation is much like the Finder... double-click on an object to open it. Groups open in the same window; datasets open in a new one that looks very similar but with the list control replaced by a grid. Attributes are accessible through right-click, or through the main menu bar.
Adding visualization was almost comically simple. The good folks at the Matplotlib project provide a ready-made control that integrates with wxPython! The code to add a plotting window was literally a few dozen lines.
Here's what the current version of HDF Compass looks like. In between, we also switched to the LGPL'ed KDE Oxygen icon set, which looks much more spiffy:
HDF Compass just had its first public release, and is now an open-source project from the HDF Group. In addition to HDF5 files, it can also handle other formats like OPeNDAP resources, using the same familiar interface.
If you've used the HDF5 format in the past, I strongly recommend giving Compass a try. Open-source development now takes place on GitHub along with a number of other great projects from the HDF Group.