Becoming Mindful

Pair programming seemed to limit control to solve problems in my own way. One of the things I love about programming, and abstract problem solving, is that there are few limits. In my head, pair programming was intrusive and disrupted creativity. It is now becoming clear that the opposite is true. Instead of solving problems our own way, we should strive to solve problems the best way. Sure, you might think you know your hacking tools in and out - but how do you know there isn't some trick to improve your workflow that you're missing?

How do you know the process you use to solve various types of problems is the best? If you have experience, and especially if you were lucky enough to work with a great mentor, you may have picked up some problem-solving tactics. Can you list the tactics you use to solve a problem? I've found that we sometimes are unaware of our own problem-solving process. Next, have you ever spent time testing out the effect each of those tactics has on the outcome of solving problems? That's a lot of data for one person to gather. Not only would you need to record all the observations, but you would need a lifetime to generate enough samples to draw any conclusions.

While conducting a research study to determine what tactics are the most crucial for solving different problems is a tremendous challenge, we can approximate the findings by being mindful and observing others. Each of us have built up an arsenal of tactics, our tricks of the trade, and you can get a snapshot of any engineer's strategy by watching him solve a few problems. How does he start: pen and paper, whiteboard, or straight to code? What next? Writing a test, defining interfaces, or taking a walk?

Becoming mindful of our own and others’ problem-solving strategies is a process that likely never ends. Thankfully, we have mentors, peers, and students to point out steps we take without notice. Consider a student seeing something for the first time, asking why you perform a particular action; or a mentor opening a dialogue to challenge a misstep.

Tight Loop

Sharing ideas and intent with other programmers can spark design discussions that lead to clarity and solutions. Great, can't design meetings and code reviews accomplish the same effect? Sort of, but not efficiently. Design meetings work well for only the broadest discussions, the details that begin to drive the design are mostly unknown until there's an implementation. Interlacing design discussion with writing code permits a higher fidelity in the design.

Similarly, distinct code reviews are useful but frequently occur too late. Peer code review should be interactive and occur as code is written, more time is saved when poor decisions are detected early. This prevents scenarios where large commits, perhaps a full day's work, is based on misunderstandings or bad engineering. When pairing, the partners should be thinking out loud. That verbal communication can prompt a design review to occur before any code is ever written. However, actual reviewing of code is also important to catch any assumptions or bad choices that are not verbalized.

Dual Modes

This hypothetical benefit of pair programming excites me the most. When people talk about having awesome pairing experiences where the pairing team has momentum and flow, I think they are taking advantage of what I call dual modes. In most pair programming literature this concept isn't given a name, instead they talk about two roles: driver and navigator. The driver is whomever has the keyboard and the navigator is the other. These names are deceptive, they imply there is only one active role, the driver, and then some pansy navigator trying to follow the action and pointing out spelling mistakes. Sounds lame.

If either of the roles should be considered more active, it should be the one not banging on the keyboard. Instead of a driver and a navigator, there should be a grunt and a commander. The grunt is trying to figure out how the task can get solved given a limited set of assumptions. Those assumptions enable the grunt to not worry about the big picture and focus on only implementing the current task in the best way possible given the assumptions. The commander is the mastermind, she should be providing the assumptions, revising them based on reports from the grunt, and watching the grunt's activity for any signs that an assumption is invalid. The importance of the navigator/commander role cannot be overstated.

Aside from poorly-named roles, much of the pair programming literature also places too much emphasis on role definitions. My hypothesis is that the real power of pair programming is two people working on the same problem but operating in complimentary mental modes. The exact responsibilities of each partner will shift throughout a session.

To Be Continued

I'll be pairing with a few people over the next month and will try to return with a follow-up about the experiment. If you've had an experience with pair programming, please share!

Permalink | Leave a comment  »

Social complexity is the study of social phenomenon through the lens of complex systems.  This often includes building computational models of social behavior, some of which are related to computer science topics such as: cellular automaton, genetic algorithms, or neural networks.  Social network analysis is also related to social complexity as it is used to quantify relationships, roles, and organization amongst agents.

• Sante Fe Institute
• CASOS at CMU
• Center for Social Complexity at GMU
• Center for the Study of Complex Systems at UMich

A few books of interest:

• The Evolution of Cooperation, by Robert Axelrod
• Generative Social Science: Studies in Agent-Based Computational Modeling, by Joshua M. Epstein
• Growing Artificial Societies: Social Science from the Bottom Up, by Joshua M. Epstein and Robert L. Axtell

Two notable blogs:

• Zero Intelligence Agents, by Drew Conway
• Three-Toed Sloth, by Cosma Shalizi 

This is a brief post that I hope to extend later.  Please feel free to make suggestions via comments, email, or twitter.

(Figure source: Wikipedia)

Permalink | Leave a comment  »

Distant view of a network of tweets based on shared retweeters.  This is shortly after the earthquake in Haiti earlier this year.

Permalink | Leave a comment  »

A few months back I started working on a news classifier for Highput. I had originally planned to use Clojure to write Hadoop jobs, but found that most of the input data was available as a stream and the allure of continuously producing results was too much. RSS feeds, twitter streams, and collaborative ranking (e.g, HN, reddit) are continuous, the classifier should be as well. The data retrieval, extraction, transformation, and loading (ETL) was done using a distributed workflow built from three simple components: Python, Beanstalk, and Tokyo Tyrant. The producer/consumer processes are Python programs, Beanstalk is a message queue and divvies jobs to consumers, while Tokyo Tyrant stores intermediate results.

Python is a great programming language for getting things done: it has loads of libraries, pseudocode-like syntax, and keeps simple things simple. Aside from gaining throughput performance by distributing across multiple machines, one of the advantages of using a workflow for data processing is the reduction of complexity that comes from dividing a larger problem. Each processing step in the workflow was implemented as a separate Python program. Each instance of the program requests a job from Beanstalk, does its work, then throws the result into a different job queue. Any of the processes can be killed at any time without losing the task. That's because Beanstalk keeps track of jobs, even after a worker has reserved it - if the worker fails to report back then the job will be returned to a ready state and made available to other workers. Beanstalk also supports persistent queues, protecting against losing jobs if the beanstalkd process dies. Finally, there is Tokyo Tyrant, the network interface to Tokyo Cabinet. Tokyo Cabinet received deserving attention in 2009, it's the best little database, supporting multiple modes useful in various circumstances: simple key/value pairs, fixed-size arrays, B-trees, and schema-free tables. Ultimately, the news classifier in production would use something with an automated process for balancing storage nodes, but Tokyo Cabinet is great for prototyping even if it's not the longterm solution.

With these tools, a distributed system prototype for ETL can be built in a day. And not just a hacked up system that should be thrown away, but one that is nearly good enough for production - add deployment management, failure detection and recovery and it's complete. That is significant work, but all are concerns for users of most heavier distributed system frameworks too.  While this solution is ad-hoc and inappropriate for some situations, it's amazing what can be done with a small set of well-designed, task-specific tools in a short period of time.

Permalink | Leave a comment  »

Been working on a logo for Lightpost Software, think it's getting closer... Have a couple other layout and color variants, but this is my current pick.

Permalink | Leave a comment  »

Or, alternate coloring:

Hint: it's supposed to be a lightpost.

Permalink | Leave a comment  »

A group of developers have started WebFinger, an open source project to revive finger by implementing it atop HTTP.  Although finger was historically used for supplying contact information and personal news, similar to personal blogs, a new webfinger + online storage gives users the ability to consolidate their online persona and retain ownership of content such as photos, videos, blog posts, and personal opinions. This is a step towards reclaiming personal ownership of online content and not filling out the same personal information and preferences for every site or social network platform.
 
Decentralization and privacy while leveraging collaboration tools such as social networks needs to be the future of the Internet.

Permalink | Leave a comment  »

Got to play with a Reactable in Chicago's Museum of Science & Industry.

Permalink | Leave a comment  »

Earlier today on #clojure, there was a brief discussion on how larger (including infinite!) arity functions are implemented in Clojure. An example similar to what started the discussion:
(apply + (range 100))
 
The interesting bit happens in RestFn and how the compiler lays out the bytecode.
 
The implementation of + is:
(defn +   "Returns the sum of nums. (+) returns 0."   {:inline (fn [x y] `(. clojure.lang.Numbers (add ~x ~y)))    :inline-arities #{2}}   ([] 0)   ([x] (cast Number x))   ([x y] (. clojure.lang.Numbers (add x y)))   ([x y & more]   (reduce + (+ x y) more)))
 
There are four implementations of +, which gets used depends on the number of arguments provided. The + function is a RestFn instance and it's applyTo method is called from apply.

So far we have:
(apply f xs) -> f.applyTo(xs).  

This then calls: 

 f.doInvoke(xs.first(), (xs = xs.next()).first(), xs.next())   

Which matches the implementation that handles the parameter list [x y & more]. The Clojure compiler, when emitting the bytecode for +, places the implementation defined for [x y & more] under the appropriate doInvoke method and overrides RestFn's default implementation of throwing an exception about an unsupported arity.

Permalink | Leave a comment  »

I just spent a good hour searching for a way to left align a couple lines of type definitions with amsmath.  The matrix environment, which provides matrix formatting without delimiters, is a solution.

Permalink | Leave a comment  »

If you’re looking for a way to use LaTeX from inside Inkscape on Mac OS X, grab [1] and copy textext.inx and textext.py to Inkscape.app/Contents/Resources/extensions.

You’ll need pstotext and pdf2svg as well, both are provided by MacPorts.

Now you should have a “Tex Text” option under the “Render” selection in Inkscape menubar.

-
[1] http://www.elisanet.fi/ptvirtan/software/textext/index.html

Permalink | Leave a comment  »

After sharing the intro song to Aqua Teen Hunger Force Colon Movie for Theaters with friends via both a YouTube link and playing the DVD I noticed that friends who received the link and watched on their own tended to enjoy the content more than the friends who watched it from the DVD.

There are many explanations for this, but one possibility is that due to the poor quality of Internet media in the past, people will pay closer attention to content delivered on that medium. The extra attention would mean viewers catch more details during the first viewing than they would if watching on TV/DVD.

Permalink | Leave a comment  »

Tracing vfs_context_proc on Mac OS X.

Generated with a nodebox script and dtrace script I wrote for a paper.  The layout algorithm was handwritten and the coloring hacked together from some existing modules.

Source at http://github.com/mattrepl/topo/tree/master.

Permalink | Leave a comment  »

Tracing lwp_block on Solaris.

Generated with a nodebox script and dtrace script I wrote for a paper.  The layout algorithm was handwritten and the coloring hacked together from some existing modules.

Source at http://github.com/mattrepl/topo/tree/master.

Permalink | Leave a comment  »

Tracing ioctl on Solaris.

Generated with a nodebox script and dtrace script I wrote for a paper.  The layout algorithm was handwritten and the coloring hacked together from some existing modules.

Source at http://github.com/mattrepl/topo/tree/master.

Permalink | Leave a comment  »

The release of Google’s App Engine has caught the attention of many prospective startup founders [1]. While the feature list and comparisons to similar services are being discussed, Google’s intent is being overlooked.

Soon, any developer with a Google account will be able to develop and deploy an application that has access to Google’s resources. That is, a developer can build a Google application without needing to be employed by Google. You build it, grow it, then Google buys it.
___
[1] See Hacker News.

Permalink | Leave a comment  »

A recent post on news.YC on the decline of content quality got me thinking about the cause of what happened to Reddit and what’s happening to Hacker News. I saw unique, quirky content at Reddit become replaced by clichés and pictures of cats. And the transformation of news.YC from startup news to hacker news to prog.reddit++.

Overpopulation is a common diagnosis of the change; usually with an argument that all crowds are stupid and so, it follows, is the content they produce. That may be true, but it alone is not the cause of quality degradation. The news.YC site began life as Startup News and at first the front page changed slowly. That is, stories remained on the front page longer, probably due to fewer total number of submissions. But then, as more users joined, the volatility increased as the quality of the front page decreased. The front page was still full of startup-related posts, but much of content was repetitious or weak.

Startup News was becoming lame [1], so PG rebirthed the site as Hacker News. Startup-related content was still welcome, but the focus was now on “news interesting to hackers generally” [2]. The stated reason for the change was “we ourselves were getting a bit bored reading stories about nothing but startups” [3]. I propose that it wasn’t that users were getting bored about startups, but that we were running out of content to share and things to say.

The pool of hacker-related material on the net is larger than that of startup-related material, but the rate of quality content creation is still a limit. Users struggle to find new content to share and our quality measurements become relative.

There are fixes, but the side effects may be undesirable. Here’s one, off the top of my head: let’s slow down. Slow acceleration and low gravity of post rank would make for a less dynamic front page but allow more time for quality content to be generated, found, and submitted. This may be enough to transform Hacker News from an increasingly vapid whirlpool to a concentrated and directed information flow, a mutating hacker journal.

___
[1] This was originally “Startup News was dying”, which didn’t say much and sounded like FUD. I’m not happy with “lame”, but it conveys the subjectivity of my statement. The point is that the front page was full of smarmy business development articles, reports that people didn’t die after quitting their jobs, and recounts of random startups throwing parties or being purchased by Google.
[2] http://ycombinator.com/hackernews.html
[3] see 2

Permalink | Leave a comment  »

While working on a pet project, I was forced to make a decision about data storage. Being somewhat conservative and lazy I decided to (mis)use a row-oriented RDBMS for tagged data objects. That is, there is a table in the database where each row represents an object and I use relationships defined in other tables to annotate each object with any number of tags.

Yuck.

Google’s press release and paper on BigTable was the first time I had noticed column-oriented tables. Ever since I’ve kept tabs on alternative database systems, matching each with hypothetical problems.

What I want is a system for storing structured data, tagging the data, and performing queries based on combinations of tags. Tags don’t have to be fluffy things like “kittens” — tags can be used to signal a particular trait, such as “all posts in the current month”.

At this point, there appears to be only one open source, column-oriented database that is still being actively developed and could be used behind a web application [1]. It’s unclear whether it supports bitmap indices [2] or is capable of handling heavy web traffic, but fight off NIH syndrome long enough to determine that.

___
[1] MonetDB, http://www.monetdb.nl/
[2] bitmap indices, http://en.wikipedia.org/wiki/Bitmap_index

Permalink | Leave a comment  »

As reddit to /., github to sourceforge.

Permalink | Leave a comment  »

Arc was recently released as an abstraction layer on MzScheme. Much has been said about the design philosophy and the lack of innovation.

The most significant blunder in the release of Arc is the absence of a packaging system. People are excited to be a part of the new community and share code that could wind up being in a standard library. There’s a potential chain reaction that will now decay without easy code sharing.

Package management is an ugly problem, but there are many preexisting systems from which to learn. If the masses had to wait several years for the language axioms to be defined, what’s another 6 months for nifty package management?

Of course, the rebuttal goes something like, “PG didn’t want to, write it yourself.” Understood and partially agree, but one of the Arc founders should have implemented package management before the public release. Not because it’s wrong to punt, but because all the excitement from the initial release will have worn off by the time that a proper mechanism for sharing code is available.

Permalink | Leave a comment  »