Many thanks to Karl for the interview.

Why don’t you give us some background on the Subversion project, what your involvement was initially and what it continues to be.

Subversion actually had one of the most clear and unambiguous creation stories behind it of any project I know. There’s a company called CollabNet who provides services to clients that include things like hosted version control.

They provide hosted project management to their clients — things like a bug tracker, discussion forums, mailing lists, version control. Well for the version control component, they were using CVS. And of course, nobody knows better than CVS’s users and indeed its own developers, what a bear it can be. And they said we really need to offer something better — let’s go start an open source project to replace CVS, which is a pretty bold thing for a startup to do.

They had a sound business reason for wanting that, which was: having Subversion be open source, would mean that it could spread in the wild, and then when a customer came to them to buy their hosted service, the customer’s engineers would already know Subversion and be familiar with it, and that would be a selling point, rather than a “we’re locking ourselves into this thing that we don’t know, we’re going to have to learn it, and then we’ll never be able to get off it.”

Why did they want to do it open source from the getgo?

So they actually said that it /has/ to be open source, and there was never — as far as I know — any serious dissent from the investors or the board or the other management. You know it was, very clear that that was the way to go.

So they looked around, and at that time I had just written a book called “Open Source Development With CVS”, and was sort of known as a CVS developer and writer on CVS. So they came to me and said, do you want to start this new project. And I laughed and said, what a coincidence: I was just sitting around with my friend Jim Blandy, we were designing a new version control system, and Jim’s got this great design but even better he has a great name for it — he wants to call it “Subversion”. But you know, nothing will ever happen, because we’re just doing it in our spare time… unless, you wanna hire us to do it? And they said sure. They tried to Jim, but he stayed at Red Hat and Red Hat donated him to the Subversion project for about nine months.

So he worked on that full time, gave us the design, and got the repository layer mostly written. And then Red Hat pretty much needed him back and he just sort of, observed and occasionally chimed in with advice on the project. But we quickly got, like, almost immediately, we had 10 to 15 very highly qualified volunteers — people who had been looking for a version control system to replace CVS, and they just sort of said, “okay, it looks like this is the one, let’s get on board”.

And I think really within a year — I could go back and check but I think we had 30 people and now we’ve got about, I’d say, 50 some global committers, of whom at any given time 20 to 25 are active. And we get lots of patch contributions and we have a lot of partial committers that maintain like, side scripts and language bindings and things like that.

That’s something that kind of amazed me when I was looking through the CollabNet visualization of projects — well first of all it was interesting that you have far and away the most commits of anyone.

Oh you mean me personally, in the project? Yeah well maybe that’s because I keep reverting my mistakes. [chuckles]

[laughs] I also noticed that there were more than 100 committers.

Oh yeah — if you count the partial committers as well. We’ve got this division in the project, pretty much the only formal division we observe, which is, there are people who have global write access — you can commit anywhere. And then there are people who, you know we know that they can maintain the Finish translations, or the Python language bindings, or something, but we haven’t seen enough patches to know whether it’s safe to have them committing, say to the repository code, where if something goes wrong, you know, that’s people’s data, we have to be careful there.

There’s no technical enforcement of this — if one of those people did commit to the repository code, we wouldn’t revert it necessarily, we would just say, you should ask first, get approval on the patch, but it looks good so leave it in. So it’s totally socially enforced. But yeah if you count all of those committers it’s like 100-some people. And their very active, the partial committers. And many of them become full committers after they spend some time getting to know the code.

Yeah. Well that’s just fascinating. I mean, who — who manages 100 people?

Oh well, we’ve evolved some systems for self-management… that is, we — one of the things we use in fact when evaluating whether to give someone commit access at all and whether to invite them to be a global committer, is, how well do they interact with the rest of the project and use the automated systems like the issue tracker and the mailing list archives and stuff — how well do they make it so that they don’t need to be managed by another committer? And then, for certain tasks that kind of require a human to do them, we have people filling these roles, like patch manager. Basically there’s somebody watching the development mailing list all the time, for people posting patches. And that person tracks if a patch gets applied, or if there’s a discussion thread that follows and eventually rejects it and says “well, that patch isn’t quite right, or that idea isn’t good so we’re not going to do that”, or, whether people basically seem to approve, but nobody gets around to committing the patch, so in that case the patch manager is supposed to take the patch, find the right archive URL, for that whole thread, and making an issue in the issue tracker saying “here’s the patch, here’s the thread, this should really get applied, there doesn’t seem to be any objection to it, but nobody’s done it. So, he keeps things from falling through the cracks.

So, we try to pick things where it’s possible to have one person manage that space. There aren’t spheres of responsibility, it’s not like one person is in charge of the repository code, another person is in charge of the working copy code, or something like that. Everybody’s equally responsible, but also there are people who are known to be experts in say, the merge tracking code. And if you are about to commit there, and you have questions, you would just go ask them, on the list of course, but you would address it to that person.

So, people manage themselves, and when there’s a need for centralized management we try to spot it and ask a volunteer to do it.

Do you think that the Subversion project is unique from other open source project in its social organization?

It’s not unique, it is unusual in the extreme degree to which it has codified its conventions. But there are other project that have also written down how they work and have sort of, guiding documents that they keep up to date as to how they operate. I’ve sort of come to think that, sure Subversion is unique, but almost every project is unique. I haven’t really seen two that work in exactly the same way. And it’s usually some combination of their different person dynamics, because of the people involved. There can be different dynamics with the corporate sponsors — not every corporate sponsor is as enlightened and hands-off as CollabNet has been — they contribute a lot of development time, but they don’t make demands on the community. They don’t say, like, “you need to have this feature in by this date, or you’re all in trouble”. They don’t operate that way at all.

CollabNet gets it.

CollabNet totally gets it. But there are other companies that don’t get it to the same degree and yet, their projects still work –

It just takes twice as many programmers. [self-righteous laughter]

Well, you know, I feel that the is more that, there are people who don’t get as involved as they otherwise would be, because they feel like they won’t get as much influence as they would deserve, even if they did put in the time. So it’s really hard to spot that kind of disadvantage, how can you tell when someone hasn’t posted a mail to your list, right? You can’t. But those projects still work, they just have a slightly different dynamic.

And then you know there are the dictatorships, like the Linux kernel. There are the total democracies — Subversion is essentially a total democracy, there is no person in charge. There are things in between where you have module owners for different parts of the code. Everybody’s unique.

Are there any features that you- when CVS was dominant, and everyone knew its weaknesses — in fact is seems almost like, the first thing you learn about CVS is how to use the basic verbs, and the second thing you learn about CVS is what you can’t do with it. [self-satisfied chuckle]

[chuckles] Yeah that’s actually, that’s the best description I’ve heard of life with CVS– you learn its limitations very early.

And it’s funny because, I think the reason that people are so sort of, sharply aware of CVS’s limitations, is because, there is this vision in software engineering of the uses of a source control system, in terms of branches and merging — people know these models, of a branch, and a merge… a release branch. But CVS didn’t support certain aspects of that– for years. It’s kind of amazing how something like Subversion didn’t come along earlier.

I am also shocked actually, yeah.

And when it did come along it took relatively hefty sponsorship.

Yeah, although in fairness, it should be pointed out that more recent version control systems have not required the same level of investment, in terms of sponsored developers, and time. Like Git got up and running in what, 4 weeks, or something like that. I’m sure it was buggy and had all sorts of weird user interface stuff going on at that point, but it didn’t take as long to get up and running– and neither did Mercurial, for example, which does have some corporate sponsorship, but it didn’t take as long.

But part of the reason that Subversion… so Subversion has a weird dynamic. I don’t know why it took so long for something to come along and replace CVS, but when something did — the first such something being Subversion — we decided to preserve as much of CVS model as we could, which meant that we had- it was good in once sense because we had a design document in the form of an existing implementation. We said “we don’t want to be worse than this, and we want to support all the major features that this thing supports”. But it also means that you can’t say you’re done, you can’t call it 1.0, until you’ve matched a certain set of features. Whereas other projects were free to go decide where their 1.0 is. Or at least decide when it’s usable, and what they can claim. We had set the bar at CVS very publicly, and so, it took us four years to get to 1.0.

Are there features that you envisioned Subversion having, that it still does not have today?

Oh yeah. The new release — this is going to sound like advertising, but it is the answer to your question — the next release coming out is 1.5, and it’s going to have merge tracking, which is something that everyone’s been wanting for a long time. It’s gonna make maintenance of release branches and experimental branches and stuff much easier. Instead of having to remember all these URLs and remember what you merged in the past, you just sort of like do `svn merge -g [branch]`, or you know, you’ll be in your working copy, so you’ll just do `svn merge -g` and the right thing will happen — um, I think I’m summarizing the feature correctly but you know, maybe there’s some more stuff there.

So, like that’s huge. and that’s something that we wish we’d had in 1.0 but, it wasn’t part of CVS we were like, do we do this or do we ship it? Let’s ship it, we can work on it later. There’s another feature, something that’s similar to CVS’s modules, the ability to do selective filtered checkout where you only get some of the subdirectories, that’s going to be in 1.5. So, yeah there are still features. And there are things we get asked for on the users list all the time, like the ability to give a human-readable label to particular revision numbers. People want that all the time.

What are you going to call it? A label?

They call it labels, I guess we would do some searching.

That’s very similar to what CVS does, as opposed to the convention, with Subversion, of the tag methodology.

Subversion’s tag and branch methodology, with a couple of exceptions, is functionally equivalent to what CVS does, it’s just more efficient, on the server side. It takes up less space — it’s very complex to explain what the tradeoffs are, but basically it’s the same thing, inverted along a different axis. But the labels thing is different because, CVS doesn’t have the concept of atomic commits or numbered- individually identified trees. So there’s really no- the thing that people are proposing be labeled in Subversion doesn’t exist in CVS. It wouldn’t be their label [tag?], because you can’t say “the revision tree rooted at revision 2614″- that isn’t there in CVS. In Subversion that tree does exists, it’s just a question of if you want to call it revision 2614 or “My Special Golden Release”.

Well in CVS’s labeling– it is called label isn’t it?

It’s called branches and tags.

It’s called tags, yeah– once you tag something, that tag can’t be modified, which is different from the Subversion-

Ah- that’s actually not true.

I mean the code in the tag– the code that the tag refers to can’t be modified.

Well, it is true in both systems that code once checked in, cannot be modified. But the tag – that is, what the tag refers to in CVS, /can/ be modified. And you can’t tell whether it’s been done or not.

[exasperated chuckle] Okay.

So the difference is that in Subversion– in both CVS and Subversion, you can modify what a tag refers to, but in Subversion you can tell that someone’s done that. And in CVS you can’t.

That’s interesting. I only used– I did not use CVS very much before I started using Subversion. So actually I’m glad this came up. Where did the convention — the branches and tags copied from trunk convention — come from in Subversion? Was that something that you envisioned as being, you know, a sort of obvious implementation of that software engineering concept, or did that kind of emerge out of something else.

That was the original insight that Jim Blandy had about — or I think he was talking with someone who’s name I can’t remember now, about CVS and its problems. And, basically the conclusion that they came to — this was in like 1998 or 99 or something was — the problem with CVS is that it’s indexed along the wrong axis, which is — it’s supposed to be taking these snapshots of things that change over time — so you’ve got a tree of files, you make a change, now you’ve got a new tree — but it’s indexed along the files, and so you’ve got lots of little changes and you have to work really hard to collect them into some coherent change at a new version of the tree. And, what tags and branches in CVS do is they attach the same label individually to a particular revision, which could be different, in each of thousands of files in a repository. And that means that finding out what a tag represents, or even indeed creating the tag, is an O(n) operation on the number of files. And it’s related to how many different revisions are in that file and how big are they.

So his essential insight for Subversion was, what’s the difference between a tag and a branch anyway? Nothing. You can’t tell a tag from a branch, until you start committing on a branch. And then you know, “this isn’t just a tag, because I can commit on it”. So why not just call them all copies? That is, what you want to do is, you’ve got a tree of files and directories, and you want to make a new copy of it, and then you want to make changes in the copy. And, if you don’t make changes and you just make the copy, that’s a tag.

Unless you need to update the documentation because you forgot to. [nervous laughter] Which I’ve done many a time.

Oh well if you didn’t branch the documentation too [laughs].

No like if I forgot to, for example, change the version number, in the documentation, in my branch, before I tagged it. So I’ll jut modify the tag, I’ll cheat.

Oh yeah yeah, I’ve done that too. Don’t look to carefully at the history of the Subversion repository. [laughs] But the thing is, if you wanted to protect tags against that kind of playing around, you could. It’s very easy to make a pre-commit hook script that watches the tags directory and doesn’t allow anything but copies of trees from trunk. It’s just that people don’t generally bother to do that because, you can always go look and see whether your tags directory has been played around in.

And that seems to be an interesting aspect of working with Subversion, and something you touched on previously– this sort of social aspect of managing code. There’s such flexibility and confidence in the ability to manage the code exactly how you want, that you know, you don’t care if you give someone commit access and they might not behave great, because you’re looking at every commit–

I think there’s kind of a deep lesson there, actually. The reason authorization protections are necessary, ever, is when you can’t trust somebody to modify something, and can’t be certain that you’ll find out about it later. If you /can/ be certain that you’ll find out about it later, then that auditing ability is enough, and it doesn’t really matter, whether you can trust a person, either for their competence or for their intentions. Because auditability and revertability mean you’re safe. So a version control system is kind of a way to implement social controls. It makes further control unnecessary.

Subversion has great… if you know what you’re looking for, it’s easy to find the information you’re looking for with the log command. But the log command is also a little clumsy. But there are tools which have come out, to kind of compensate for that, and I think those tools, are easy to make, because the API to Subversion, is very, sort of, complete. So for example, Trac. Trac is this like, everybody’s favorite sister application with Subversion, and some people, myself included, could not imagine– I always set up a Trac system, to watch all my Subversion repositories, because it’s just such a nice visualization of the timeline, viewing changesets, the history of different files–

I love Trac, I also have used it on some projects, or have participated in projects where other people were using it. If I can gloat for a second– I feel like that example, that phenomenon in general, are kind of a complete vindication of the emphasis we placed on APIs. And part of the reason we wrote it in C even though C… makes you do more work for the same results, as compared to like Python or something, was we wanted to have binding surfaces to lots of other languages. And so people came along and they they used those APIs and they exposed them to Perl, to Python, to Java… there’s somebody doing Scheme now. And you get programs like Trac which, can add features using Subversion as a substrate, that we would never have thought of. And that was exactly the goal, in making those APIs. And it totally happened, exactly as everyone dreamed it would happen.

Is there a uh, is there kind of a layer of visualization— for yourself, when you use, for example, the log command, are you ever wishing for more interactivity? Is it something where you think “yeah, Subversion provides this flexible API and I’m glad this ecosystem of tools are there” or are you ever like, “uh, you know, like, there should be a way to take a different path through this information, with Subversion itself”.

Well, I think I might have that sort of developer’s weakness of knowing my tool too well, and so I think of– when I’m thinking of a problem, solutions come in the form of weird ways to use the tool, that you kind of have to be very familiar with it to think of. So I don’t find myself craving those other ways of looking at log data, or visualization tools. I’m also not a particularly visual person. But I know that most users are, and it’s important to support them.

When I said “visualize”, I really just meant– I guess I meant filter, actually. To get to the data that you’re looking for.

Oh– yeah, I’ve even written a filter that takes a stream of log data, and a regular expression, and just returns the log entries that contain that regular expression, that kind of thing.

That wouldn’t happen to be for, seeing what you’ve merged into another branch, would it be? [chuckles]

Actually no, but, well, you’ll need that less with merge tracking [in 1.5].

Right, and that’s what I was going to say, maybe that’s another reason that you weren’t too concerned about this is because, I think that’s one of my primary frustrations, is when you have to do svn log, stop on copy, and then peer or filter through things to determine what has been committed to the branch.

You know what– now that you mention that, I realize that– so the way I run svn log normally, either I have the file of all the log data sitting around or I run it, in a shell, but my shell is run in an emacs buffer, and so I might get 20k lines of log output but then I can do incremental search and regex search, backwards in the buffer, because it’s all inside Emacs, and so actually I am doing all of my logs, inside an extension tool, I just never consciously realized it before. And I’m doing stuff like what you’re saying. So, I’m using Emacs to compensate for a lack of a feature in Subversion. I just never knew it.

So, let’s talk more about the new features in 1.5. I guess lets start with merge, which seems to be the primary feature.

So we had this kind of very vague handwavy vision for years and years that we would one day keep track of what changes had been merged from one line of development to another, like trunk to branch, branch to another branch, branch to trunk, whatever. We would do that using Subversion’s properties, which are metadata — key-value pairs — that you can attach to any versioned object. So, files can have properties, directories can have properties. And so we’d have this one called svn:mergeinfo.

Like for example if you merged changes, revisions 2000 through 2010 from trunk into the branch, then the mergeinfo on the branch would record that fact. And then if you’d asked trunk “merge me everything you don’t have, into the branch”, it would not do those revisions, because you had already done them. And then the idea would be you could ask the branch — you could write tools that would ask the branch — “has the bug fix for issue #37 been addressed?”, and your tool presumably could go to the issue tracker, or ask some other database, “which commits in the repository addressed issue #37?”. Okay that’s revisions 2000, 2003, and 2009, and then it would go ask the branch “do you have revisions 2000, 2003, and 2009 merged into you from trunk?” and the branch would say yes or no. And then you would just be able to answer with the press of a button “yes, that bug is fixed on these following release branches:…”.

Not all of that chain of steps is in Subversions, but the asking a branch which revisions have merged into it, that is now in Subversion 1.5. And so the rest of that is for tools like Trac to implement, and they now have the APIs to do that. So I expect to see it very quickly.

So I imagine that this merge tracker will be able to skip over, sort of, spots of things that have been merged into something, and then merge in the rest. That’s the idea.

Right. So one thing it does, and it’s very important that it be able to do this, is you can cherry pick merges from various lines and bring them into some line, and then you can just say “okay, merge everything you don’t have”, and Subversion will figure out — I’m gonna get killed if I’m summarizing this wrong, because I’m not actually working on this part of the release, but I /think/ this is correct — Subversion will go find all the stuff that hasn’t made the discontinuities where necessary in the merge ranges, to compensate for stuff that’s already been merged, and successively merge the remaining stuff.

That’s great.

Yeah. I mean it’s exactly what you want. We actually had a really thorough requirements solicitation process around merge tracking. CollabNet sponsored several meetings with a bunch of customers of CollabNet’s who use Subversion at a enterprise scale. And they described in great detail what they needed, they made use cases available to us, and then we took the results of that to the public lists, and said alright folks what do you think of these, and add some more cases. And spent a long time polishing it up into a functional specification. And that functional spec has guided all of the Subversion developers, in developing this.

So, I think it’s, that to me is a classic example of how a free software project and a for-profit corporation that’s sponsoring it can really take advantage of the fact that the corporation has customers, and access to information that the project might not otherwise have. So, I don’t think for example that, say the Git project — which is doing great things, and I’d love to get a chance to try it out myself, but they have one highly unusual customer, in a sense, which is the Linux kernel community, and as far as I know there isn’t a lot of input from corporations that have thousands of developers, not all of whom are extremely familiar with their tools, telling it here’s what we need from say merge tracking, or something else. So it’s a much more ad-hoc requirement solicitation process. Subversion tried to have a very thorough, formal, well-described process, and I think the merge tracking we’re getting is going to be a result of that.

Something that, well, my primary project is downstream from another open source project. And I actually use SVN’s merge tools for that.

Like the svn load-dirs script? Or the svnmerge.py?

Nope, I just use svn merge.

Wow, okay…

REDACTED – tedious 10-minute tech support session.

[chuckles] The Subversion project discussion turns into the Subversion tech support discussion — yeah, it always happens.

[chuckles] My more general question was, are there any features that you’d like to see, with, sort of smart interactions between different repositories. And I guess you kind of answered that in saying that the metadata is there, and it’s awaiting use.

What i’d really like to see, and what some projects are starting to do is, version control systems being able to talk to the repositories of other version control systems. And I think like, there’s a Mercurial thing that can talk to Subversion repositories now, and, Git might have that, and, in some sense we’re not doing our part, because I don’t think we can extract easily from Git or Mercurial repositories. But there might be some sort of in-between thing that can sit there and translate.

[smart alecy] Oh you’re saying your API is better. Yeah, I hear you.

[not getting my awesome joke] Well, you know our APIs our great, but they’re Subversion specific.

But it would be nice if… if the semantics of the different systems line up well enough, it would be nice if which one you use becomes a matter of personal taste about clients, instead of this choice that has to be made for the whole project. But we’re not at that level of abstraction yet.

Yeah– although– I think all of these SCMs kind of, endeavor to be semantics only, right? To a certain extent. It’s kind of like — that is the fundamental thing that differs between the SCMs, is the verbs — the actions. So like, I don’t think that, that you can be seen as kind of a syntactic layer.

Well, it’s true that those are the differences. But even with that they’re still very close to each other. Here’s a perfect example: Subversion is considering taking the Mercurial revlog format and using it as our repository format, in a future release of Subversion. So, that wouldn’t be possible of the semantics weren’t basically the same.

Very true, yeah. What’s your take on distributed version control systems… where they have a place, where they don’t have a place.

Uhh, complicated. I really like them… well, i should say, I love the idea, I think it’s, it’s the way I would want to work. Although for historical reasons I’m using Subversion most places and will continue to I think. But if you’d shown me a distributed system like 10 years ago I probably would have started using it right away. But, I don’t think they’re the answer to everything, even though it is formally true that they are a superset of centralized systems. There’s nothing a centralized system like Subversion can do that a distributed system couldn’t do. But the difference is whether you’re working with the grain or against the grain of the system, in a sense.

Well so, Ben Collins Sussman, one of the other founding developers of Subversion, has a great quote that he asks about the distributed systems, which he also is interested in and you know, thinks that there’s some good stuff there. He also is in a position where he’s been talking to a lot of enterprise users of version control over the years, and when it comes to one of these new distributed systems he always says “Looks great. How’s the corporate rollout going?”. Meaning, that’s great but, if you try to get 10,000 developers of some huge company worldwide to wrap their heads around this idea of like, everybody’s their own repository, and you push and pull changes randomly, and you can all decide to publish over to this central repository for a while and then you can merge that into another one, it’s not gonna fly. It’s too complicated.

And just because we as revision control experts who spend our lives on this stuff can find it interesting and very usable, doesn’t mean that users are gonna find it that way. And it’s funny– the emergence of these systems has made use realize in Subversion like, what are our real strengths? Simplicity. Comprehensibility for users who may not even be programmers, let alone revision control experts. And centralization, paradoxically.

So for example, What are the most important features in Subversion — or the ones that have gotten the best feedback from the user community? Well, locking. That is, the ability to tell the repository “this following file is locked”, and then when somebody else tries to edit it, they get a little bounce-back notice saying “so and so has a lock, you better talk to them first.” It’s the total opposite of a distributed system. You know, it’s the ultimate centralized feature. But it’s one that everybody wanted.

Another example would be — that I hope we’re gonna get it, not in 1.5 but in 1.6, is log message templates. You know, formatted things that you can use to guide your committers as to how to write their log messages for each commit. That again requires everybody to be consulting a central repository regularly, to get updated templates. So, I think we’re starting to see that the centralization — it is in some ways a weakness, but is is also Subversion’s strength. It’s the thing that is most attractive and usable for corporate version control users, and this s kind of an edgy thing to say, but I think open source software that is useful for corporate developers has better survivability characteristics than software which doesn’t attract them so much. Because we get funding. Subversion is never going to have to worry about resources. And CollabNet is not the only company that is spending money on Subversion development now.

So, I kinda feel like, having features that put your long term survivability on a firmer footing — they’re sort of — they’re innately valuable, somehow. In an evolutionary sense.

Okay, I think that’s all I’ve got. Thanks Karl.

Thank you.

Why don’t you quickly tell us about TextMate, what it is, and how long it’s been around.

TextMate is a text editor used for structural text — that means programming languages mainly — but it also has a lot of usage in the LaTeX community and Markdown and stuff like that. It has existed for roughly three years. What sets it apart is probably that it’s very easy to customize. Similar to Emacs and vi it can be customized, but unlike Emacs and vi, at least in my humble opinion, it’s very easy to do these customizations, and it’s also very easy to distribute the customizations. That has really built up a community around the text editor, which means it’s suited for a lot of tasks that I haven’t really developed the program for, but which other users have developed it for.

Emacs and vi have been around for decades, and they have their own userbases of people who like to customize them their own ways, and then there are all the language and platform-specific IDEs. You got into the text editor business, where it’s not obvious that there’s an opportunity in that field where you can make everyone happy, but it seem like you’ve hit a sweet spot. And now that you’ve done it, it seems kind of obvious — I can’t imagine programming without TextMate. When you started making TextMate, how certain were you that this sweet spot was there, and how did you envision TextMate fitting into that spectrum?

I did it for myself, and I was using Objective-C at the time, and that’s sort of the market I had in mind — people like myself needing an Objective-C editor. I was rather pessimistic because we have Xcode which is one of the IDEs built for Objective-C. And I met David [Heinemeier Hansson]– he was doing Ruby on Rails at the time, which wasn’t public at the time — and he needed an IDE for Ruby on Rails. I didn’t want to hardcode the editor for a specific purpose. At the time I did see the potential in making things customizable, so rather than write the features David wanted, write it so that he could add them himself, and I could add my Objective-C functions myself. So I did have that in mind, that it was supposed to be customized for the different domains, but I had no idea that there was actually that many domains out there — something which became apparent very quickly because I think it was a month later and there were bundles for Lisp, Erlang, all sorts of languages which didn’t have a dedicated IDE. So initially it picked up all the languages that had no IDE, and later it actually started to compete with the languages that did have an IDE. But I didn’t foresee that, not at all.

When did you meet David?

Our friendship goes back to a computer copy party in 1995– are you familiar with the term copy party?

No.

It goes back from before the internet. We actually met with our computers and floppy disks, and then we copied software because we couldn’t get it from the internet– that’s the origin of these copy parties. They quickly developed into demo parties, where you had demo competitions. So i met him in 95 the first time, but we didn’t really become friends at the time. But he became friends with some of my friends, and I met him again I think in 2001– he had some RAM, I had just bought a mac, and my RAM didn’t work in my Mac –

*laughs* Are you serious?

*chuckles* yeah. So we met at a mutual friend’s place, and I had this RAM which didn’t work in my Macintosh, and he had some RAM which did. So he offered to switch, and I got back to his place, and I saw all these books in his bookshelf, and I was very impressed, and thought “oh you read all this literature? You’ve really developed since I met you like, 6 years ago” (I didn’t say that) but *chuckles* that was my thought. So we hit it off, and we have been regular friends since then.

How early did he start telling you about Rails?

From the start — he told everybody about Rails, so of course that included me.

Did he think that Rails was going to be big back then?

I think he did. I think he’s… he’s very good at marketing, and I think he knew that he’s very good at marketing, so I think he knew that he did have something good back then, and if he was able to market it successfully, it would be big. So I don’t think he ever imagined it would be as big as it is now, but I do think he had an idea that it would be big. And i saw him at Roskilde University Center — that’s the first lecture he actually gave on rails (it’s in Danish, so probably few have seen it) — but he really did a very good presentation back then, and I was — and I don’t do web development stuff — but I was swayed after seeing that.

Is TextMate today what you thought it would be when you started it, in terms of functionality, or popularity?

Neither, I think. Well actually if you take functionality first, I’ve always seen TextMate as an ongoing project; I know that there won’t be a time when I’m out of ideas, things can always be better. But the functionality I initially had envisioned was like, a parallel to the current development. I hadn’t really realized the power of the declarative aspect — the semantic aspect — of the scope selectors; how much power that actually gives. 2.0 will be sort of a redesign. Well, not a redesign, but the entire infrastructure will be much better. It’s difficult to say this in few words; it’s the same featureset as 1.0, but in a way where I actually know that these are the features I want, so it’s designed better, so things fit better together. That’s a long way of saying: no, I didn’t really have a roadmap.

So you didn’t realize how powerful the scope selectors would be.

No not at all. So [in TextMate 2], more will be based on scope selectors, and more will be based on context. Currently, a scope selector is just a way to pinpoint a context, but the context presently is limited to where you are in the document. You can’t select which project you are in. Am I in a standalone Ruby file, or a Ruby file in a Rails project? That’s also the context. Am I in a Ruby file which is under Subversion control? That’s also the context. [So, compared to TextMate 1], there is a lot more context which the scope selector should also be able to select out. Even context like, is there a selection in the current buffer when I press this key? That can also affect things. Those are things I really realized in retrospect.

I see things kind of creeping into the bundles — more and more sophisticated use of scope selectors if you hit “show scope” you sometimes see 4 or 5 levels of scope — for example SQL strings in PHP are scoped as SQL and are highlighted accordingly, but if you have a PHP variable in there, that’s scoped properly too — it’s just gorgeous. It’s a very sophisticated interaction one has with a text document. When you’re programming you know that the text has all this meaning. When you become a programmer, and as you become a better and better programmer, you become more aware of the different levels of meaning in a text document, and I think TextMate really brings that out.

And the more TextMate knows about your document, the more functionality you can actually hook up to it. For example commenting stuff– no matter which context you are in the commenting stuff will always do the right thing, as long as TextMate understands which context it’s in.

And how does TextMate’s popularity compare to your expectations when you began building it?

My goal, initially, was to break even. I was like, I’m doing this for fun, and in the worst case, I will get a good editor out of it, and the case I hoped for was to break even, so I could make a living out of it. And, it has gone a lot better than that. But I’m a pessimist. *chuckles*

The extensibility of TextMate is very obviously a lot of its power. I think a lot of people who make software, whether they are big companies or independent developers, often think very hard about how extensible their product is going to be, because, if a feature is missing, then they can sell it to the user in a future version. So, with a product as extensible as TextMate, you can’t sell those features to people. They can either make them themselves, or your bundle community is making them. What’s your attitude towards that, and how confident were you that you could still sell the product even though it was so flexible?

I think that there are always features one can add, as I said earlier. And the customization infrastructure, that can also be vastly improved. That has never been a concern — on the contrary. The fact that people can do the features themselves, or get another person’s bundle, that’s great, because then I can spend the time on the things I think are important. Rethinking the infrastructure, or being obsessed with Objective-C for example. For example, I don’t do HTML, if I ever do stuff for the web I mostly do Markdown or something like that — and people write me and say “Wouldn’t it be great if HTML mode had this and this and this…”. And I can say “You use HTML, tell me”. *chuckles*

So you can make an editor that is good for every language, but you don’t have to know every language yourself.

Yeah. And actually, a very cool thing, about this sharing of the customizations, sometimes I do have to build something in PHP or Rails or some third language, and then I start to learn how many neat things there are in TextMate for that particular mode. Like I had to do a presentation in LaTeX Beamer, and I had never used Beamer before, and it was like, when I do ‘frame’, it completes it, and when i do ‘itemize’, it adds in which slide the item should appear on, and it’s like, “I don’t have to read the manual for this language.” *laughs*

What kind of relationship do you have with the bundle maintainers?

We have an IRC channel, and most of them are frequently there, we talk a lot together. They have a bigger chance of influencing the future direction of TextMate because I know that these guys know what they are talking about, because they are actually using the infrastructure.

How do you develop a relationship with someone, to where you can give them commit access to one of the bundles?

That has changed over time. Initially I hand-picked a few people who had been sending me bundles by email, and gave them commit access. And then I basically started to open up. Whenever somebody had a bundle for some language, he just got commit access to that bundle. But I’m starting to close down again where if you have a bundle, it will be added under the review folder, and they get commit access to that review folder, and they will get some review comments, and they have to fix stuff, and when things are in order, it will be moved to the official place, and then they can get commit access to the official place. But the existing bundles, they still can’t get access to them unless they show themselves worthy.

But I really wanna move to a distributed version control system, because this centralized thing, it puts so much administrative overhead on me, and I think it’s not ideal for the way things are working. People are sending patches to the mailing list, and then somebody with commit access to that bundle has to apply it. Sometimes one person writes a bundle, then he loses interest in that particular language or that particular task, somebody else does a lot of work with that bundle, but that person doesn’t have commit access. So, I want to move to a distributed revision control system.

How would that be structured?

I’m still not entirely sure, but ideally, each person will basically just — if he has improvements to a bundle, he will just publish it. I think I will invest in a server where I will give people all the accounts they want, for publishing bundles. But it will be under their own account. So everybody can take a public bundle, make changes to it, and publish it under his account. And then users can choose which version of each bundle they want. I will compose a list of who is doing the best work on each bundle, and those will be the official bundles included with TextMate.

But users are really free to pick whatever version they want to take. And then it becomes easy to see that– we have for example the blogging bundle, which is stale from the original contributer, but this other guy’s bundle is like, doing a lot of stuff, if I check it out and think it actually looks reasonable, so maybe I should move his bundle to be the official one. But he will still work on it on his private account, and only he will have commit access, so if somebody contributes, well they will have to contact him, or if he is not responding to his emails, then they will just fork his bundle and publish that. At least that will move all the administrative work away from me, and to the [individual bundle developers].

How do you think that open source culture, or open source patterns of development, have influenced TextMate, either in its bundle system, or your approach to developing TextMate?

Well the bundle system is an open source project with a central repository. It’s an open source project like any other open source project. TextMate itself — the only thing I have from open source is “release early, release often”. I try to do frequent releases. Another thing, and open source may have pioneered this to some extent, but I think it’s common sense — try to have a dialogue with your users, through the mailing list, through the blog. Actually read what users are writing on the mailing list, and interact yourself on the mailing list. So you can parallel there between TextMate and open source.

Well, you have a very active presence in the support of your software. and that’s something that I’ve pretty much only seen in open source. I mean you have a channel on freenode. How many commercial products have a channel on freenode? I think that’s a very strong influence from open source. And you’re always on the IRC channel! Do you ever feel overwhelmed?

Well I do feel overwhelmed, but actually not by the IRC channel or the mailing list, that’s actually great input. I do feel overwhelmed by users like, just going to the contact page and firing off an email and asking some stupid question. That’s what I feel overwhelmed about, but the IRC channel and the mailing list, that’s really no problem. I actually want to move all the dumb questions to the mailing list, so somebody else can say “this is stated in the documentation here”. *chuckles*

So you enjoy the community in the IRC channel.

Very much. And also, I’ve had lots of good TV show recommendations and stuff like that out of it. *laughs*

Do you always use TextMate to develop TextMate?

Yeah, yeah, always. *laughs* Initially I did use Xcode, and even after the release of TextMate, I think it took around 3 months before I switched to TextMate.

When was that?

Well I released it in October 2004. Chris Thomas, one of the users whose contributions I really like — he is the one doing most of the Subversion bundle — he did an Xcode build command, which did a very nice graphical output for the Xcode build process, and that’s actually when I switched to using TextMate.

*laughs* Is there anything from Xcode that you miss?

No not at all. But I haven’t used Xcode for like, three years. It didn’t have completion back then, and it didn’t have a lot of things back then. It probably has gotten a lot of new features. We also have Joachim Mårtensson, he’s doing completion for Objective-C in TextMate so I’m not really missing Xcode, no.

I’ve never done Objective-C but — doesn’t Xcode have pretty advanced breakpoint features?

I’m not a user of debuggers. But that’s what some of the people using TextMate for Objective-C development are telling me, that it should have debugger support. Not many of them, but some of them. So I imagine if you are used to using a debugger that would be something you would miss.

When you’re developing TextMate, do you do unit testing?

No. I have tried it a little but… the problem is that with a desktop application you have state, and each user action is actually just mutating the state, this global state. So it’s so darn hard to do unit tests. You can do unit tests for the algorithm, but I have been programming since I was 12, I can write an algorithm without needing to do a lot of unit tests and without breaking it like two days after. And I have tried, because there is a lot — or there was a lot of buzz about unit testing, it seems to have died out a little bit — but I have tried to apply it. But it’s like, writing the unit tests is like harder than writing the code itself, because you have this state, and then you want to simulate the user as mutating the state in all sorts of ways, and…

*laughs* I know Allan. I know it’s hard, but you’re still supposed to do it. Does DHH disagree with you?

*laughs* Oh I think he does. But I think if I bring it up with him, he would say, “Oh yeah, but enterprise programing is something else”. So we don’t really disagree, we agree that we have different domains.

How MVC is TextMate?

I would say, a lot because the Cocoa framework is really MVC. So you have your views, you have your controller, and you have your model. But in TextMate, the view and the model is strangely tied together. If you do something like, if you enable soft wrap, that’s a change in the view, but you really need to be able to break these lines, and you have spellcheck involved in that, and syntax highlighting and stuff like that. So, you have many view-actions on a model. Syntax highlighting itself is also a view thing, but it is too hard core to be in a view, it really needs a model.

It seems that it’s easier to test projects that are very strongly MVC because you know exactly where the thresholds of state are, which is certainly the case with Rails, to the extreme. Writing tests in Rails is childsplay. Whereas I’ve tried, many times, I’ve sat down and I’m like “I’m going to write a test suite for WordPress”, and like, it never happens. There are globals all over the place, which is, from the history of the PHP coding style, it’s practical in certain ways, relative to the history of the project– it’s nothing to be ashamed of necessarily. But like you said there is just so much state that you have to maintain in order to test things consistently.

And another thing, I’m using something called generic programming, which is a way to abstract the algorithm away from the data storage. And that’s something I’m very fond of. I think it’s mainly in C++ where you really have this paradigm. So, for example, I mentioned the soft wrap– something that wraps text at a certain column. We can do an algorithm for that, and that algorithm doesn’t really need to know about the storage of our actual text. And we can do another algorithm for the syntax highlighting. We can do another algorithm for the spellchecking. I really like to take out these algorithms which can be tested in isolation. You wouldn’t really call that a model. I’m not really sure what you would call it. But that’s how I separate things. I try to abstract the algorithm away from the data storage. The thing is all these algorithms actually work on the data storage, which is really something which is going to be visualized, so you could call that the view, but you could also call it the model, but if it’s the model, well, when you render the text isn’t that the view, and…

Are you considering any other projects?

*chuckles* No comments. I’m considering another project… umm.. how many will see this? *laughs*

Millions.

Actually, I’m considering a joint venture with another guy, and my hope is that I will take more of an architect guidance role, and he will do more of the coding, because I have plenty on my plate, with respect to coding. And it is macintosh software. And, yeah, let’s leave it at that. *chuckles*

Well.. is it software engineering software? Or you don’t want to say any more–

No I don’t want to say any more. It’s mostly because people will start writing me letters wanting to be a beta tester. *chuckles* So don’t write me asking to be a beta tester…

He’s making tax software. You don’t want to test it.

Yeah. *laughs* And also I don’t want the hype either.

You said you never really dabbled with web. Are there any other technologies that you like dabbling with?

Well actually I did do a few web apps, for example I did my own ticket system. Actually what’s the question, are there technologies I /don’t/ want to dabble with?

No– *laughs* Well if there are any technologies you want to stay away from, if you have some strong feelings about that…

Windows maybe? *laughs*

No like, have you ever tried out 3D, or–

Oh okay, yeah — I did a master’s thesis in computer science, and one part of the course is that you have to do a ray tracer, another is you have to do a 3D renderer, and you have to do a multitasking kernel–

So this was a very broad curriculum.

Yeah, it’s different courses; we have one kernel course, one compiler course where we have to do a compiler. So, related to that, I have been through all the various subjects in computer science, and I have used some of it since. I think the thing that probably interests me the most is probably compilers. Even though, it might seem a little boring, but it’s also very interesting…

Well I went to Rice where we have a lot of languages and compilers in the curriculum so, I think it’s cool too.

I’m actually using, for the scope selectors in TextMate and the snippets, I’m actually using actual grammars for that. I’m using ANTLR. This also means the snippets will have a little bit more syntax in the next version, because it’s so much easier to just add it to the grammar file, than actually have to code your own parser. Initially I did code my own parser for snippets but when you want to nest things, it gets really hairy.

You said you implemented your own ticket system. What was your motivation for doing that instead of using an existing tool?

I did evaluate various existing tools. One problem is that the sign up screen for filing a ticket was too complex, I figured. But the main problem is that I get, on average, at least a few tickets a day. And each time I have to log into some website and click and click and close and write some reason and close and click and reload and stuff like that. So, that was just, too slow for me. What I wanted was a mail interface. So I get an email, that says, new ticket, this is what it’s about, and if I want to, for example, tell the user I need steps to reproduce, I just wanna reply to it, and say “steps to reproduce”. If I want to close a ticket as a duplicate, or close it as some other thing, I want to reply and then just write tag:close, and then it gets tagged as closed so now this ticket is closed. I can handle everything in mail. The motivation really was that I want to cut down my own workflow. In all the systems I looked at, I had to go into a webpage and click so many times… and for me that’s enough motivation to write my own system.

So you interact with your issue tracking system–

Via my mailer mainly, yes. It is of course backed by a database. And another thing I wanted: when I release a new version of TextMate, I mention which tickets in the release notes are handled. And that automatically closes these tickets and sends out email notifications to people involved and people who have commented on these tickets. That was another integration I wanted. So I actually don’t have to close the tickets that I’m actually fixing in TextMate. So it was really just to cut down my own work.

What do think of the ecosystem of software engineering tools, like trac, and even subversion– do you use subvesion?

Yeah, I do.

Is there anything you see that you really wish was improved? It’s a very general question…

Subversion I’m very happy with. I recently started to look into the distributed systems. But Subversion works very nice, and has a very transparent model. It’s like, it’s a file system, but each change you do has a timeline, and that’s so simple to understand. The problem is that, you only do things that you would normally do with a filesystem. Where, with something like the distributed systems, you actually think outside the filesystem. You actually start to do the branches where a branch is not just copying a folder. So, that really opened my eyes, but I’m not unhappy with Subversion. It could support branches better, but maybe that’s because the model they have is not really ideal for branching in the first place. So I’m happy with Subversion. Trac, well I haven’t really used Trac extensively, but if I had to use it for a ticket system, which is the main thing I would use it for, it would have to have a mail interface. *chuckles*

Why isn’t there a damn stop button in the global search dialog box? *laughs* I just had to ask you.

*laughs* That’s only because the implementation currently in TextMate is the first implementation I wrote, prior to the first release, and I haven’t touched it since. In retrospect, seeing how long the wait for 2 has been, I probably should have redone the Search in Project, but it will be much better in 2, and it will be incremental and stuff like that.

If you wanted to add a stop button to Search in Project in TextMate 1, how long do you think it would take you?

*exhales, thinks*

I’ll pay for it.

It would probably…. more than a day. The problem is that, in desktop applications, as soon as you have to do two things at the same time, you have to involve threads, and as soon as you involve threads you have locking on shared data, and race conditions, and unexplainable crashes because of timings, so it’s really like…

Sure. [I wasn't buying it, but moved on] The bundles in TextMate are very powerful, and once you know the commands you want to use, it’s pretty easy, and if you know exactly what you’re looking for it’s very easy, because you have command-control-t…

*chuckles* Yeah.

…but it’s not very approachable, for new users–

I definitely agree

–it’s kind of weird… you click in the bundles menus, and it’s not clear how to explore what’s in there… How do you see that experience, the interaction with the bundles, evolving?

Well, I’m not really happy with the current state of affairs. The main hurdle in this is that, you have… for example in Markdown, the primary action in Markdown is probably preview. The primary action in Ruby is probably run. The primary action in C++ is probably compile. So if we want to do a toolbar, how do we do this toolbar? Do we have a run, a compile, and preview, or do we have one button with this primary action which changes. If we do this one button which changes, how do we allow bundles to say “this is my primary function”. That’s sort of the hurdle. Want I want to do, and I will talk specifics here, I want to introduce a tagging system for bundle items, so you can say, “this is the context help item”, “this is the run item”, tag it like that. And then I want to do a query language — because we know writing language parsers is cool — I want to do a query language where you can simply query the bundles, where you can say, “in this given scope, give me whatever action matches this tag”, and you can do boolean logic, where you can say “I want to run or preview or compile”, and you can use parentheses, and you can use boolean ‘and’, and you can say, for example, “give me the first match”, or you can say “give me a menu if multiple things are matching”.

That’s how I want to approach it. I want to have the toolbars, I want to have the palettes, and I want to base things on these queries. And I also want to take this to the actual current menus, because currently all the bundles are in their own separate space under the bundle menu. And some of the items don’t really belong there. For example, it’s very unintuitive that a lot of the text actions– like “strip trailing white space” in the text bundle, is under the bundles menu. Users expect that to be, maybe in the edit menu. But at the same time, I also want consistency in the menu system. But that’s the direction I’m going in.

So you’re adding toolbars and palettes. How do you think people will respond to that? Right now, the TextMate community is so– there’s not a single toolbar–

The good thing is that on OS X you can hide the toolbar. I think I will probably have the toolbar hidden by default. But then you can enable the toolbar, and whatever file you open, you will see actions which make sense for that file. For example you open a Ruby file, you will see “oh I can run this, by just clicking here”. Those are the things people currently don’t discover, but they should be able to discover with the toolbar.

Right okay, so, a highly context-sensitive toolbar.

Exactly, yeah.

Will it also change with context within the document?

Yeah, and that’s a bit of a problem I think, but on the other hand, if you are in an HTML file, and you go into the PHP part, the online help will change. Some of the time, the label will be the same, the action will just be changed. For example, you have help, and you can do help on a tag, or you can do help on a PHP function name. And even though the actual command calls for the help toolbar button will change, the toolbar button will be the same, stay the same, have the same key, have the same label.

So for example, you could have a preview. For example in LaTeX, it’s actually typeset in LaTeX, you can go into a listing environment, which is a source code listing you can have in LaTex, and if you go into a listing environment you could actually have that changed to a “run”. That’s something I still need to work out. Overall I think the concept will be much better than the current one.

What kind of palettes will there be?

Actually palettes are something I don’t really like, myself, so I will just make it like, you can do the palettes you want to do using this query language, and I don’t care, and there will be none by default.

But it’s going to be part of the infrastructure.

Yeah, it will. So I will just build an infrastructure with palettes, using this query language that I am introducing anyway, because I want do the toolbar, I don’t really want to do the palettes, but the palettes are more or less for free, and then I will see how people use them.

Well that’s all I’ve got. Thanks a lot.

Thanks.