October 20, 2005

The Parrot post-mortem


Or: Where things went wrong, and where things went right.


Parrot's not dead or anything, so far as I know. There's too much riding on it for too many people, and last I checked (which, granted, was months ago) it was still going, so this isn't really a post-mortem on the project, but rather on my tenure with it. (Which is dead) This, I'm sure, isn't all of it, but I've been jotting things down as the time's passed and filled in the explanations for the bits and pieces. Hopefully it'll help other suckersvolunteers who might run projects in the future.


Trusting the code gnomes: Good



This is something that worked out pretty well -- that is, checking in code that was barely OK, or minimally functional, and trusting that people would take the skeleton and flesh it out. This happened a lot, and it was quite nice to see.


It's not at all surprising. A lot of what we were doing in parrot was scary for a lot of people. Unnecessarily scary, but that's irrelevant -- we were doing Bizarre Dark Magic, and it didn't matter what we said, actually creating some of that from nothing was more than almost anyone thought they could do. On the other hand, it's far less scary to take a piece of working Dark Magic and change it around some, patch it up, and make it do more. That was relatively easy for a lot of folks.


It took me a while to recognize this, but once I did it was helpful, both in getting stuff done and in getting new people involved. There was a pool of "well, that stuff kinda works, but it's nasty and if it did X too it'd be great" code folks could poke at, as well as a good chunk of code that got checked in with "barely functional but works" checkin messages. Those tended to get quickly rewritten by folks watching the list that wanted to help and saw something they could comfortably work on.


Keeping an eye on the prize: Good



This was a constant fight, made worse since a lot of people didn't do this. (And in part made worse because the long-term plans were detailed in a dozen or more different spots making it difficult for anyone to keep track) Damn near everything I designed in parrot was done with an eye towards what I wanted the total package to look like. That meant juggling threads, Unicode, alternate character encodings, asynchronous IO, potentially asynchronous garbage collection, continuations, events, notifications, perl's dynamic data, all the new stuff Larry was pondering for Perl 6, along with all the stuff perl 5, python, and ruby do now.


This is one of those things that's tough for most people to manage, not because they're not capable (dunno if they are or not. I assume people are) but because to do it means having the whole design in your head, and that's tough if you're not the person doing the design. It's doubly tough if some of the design goals are things you either just don't care about, or actively dislike, and we had a lot of both going on with various people.


Skimping on the docs: Bad


This was a pretty constant refrain -- where are the docs? Well, I didn't write enough of them, and the ones I did write tended to spark fights, either because people strongly disagreed or they weren't clear enough. Unfortunately I found it tough to grab the time I really needed to get the documents written, which was a constant problem.


It didn't help that people tended to ignore the documents once they were written, which pissed me off more than once. Well over half the complaints about docs being out of date were because the code as implemented didn't match things as specified in the docs and the docs were what was supposed to be happening. (This is important -- don't whine about out of date docs or missing docs if you've gone out of your way to ignore the specifications as written)


Not getting an HLL up and running and maintaining it: Bad



Assembly language is fine as far as it goes, and having an assembler done Real Quick(tm) was damned important. What I should have done shortly after that was to get an officially blessed and maintained Parrot Twiddling Language that would be usable by anyone who isn't keen on really low-level stuff. Parrot got a lot of Perl people (and later some Ruby and Python people) but Parrot itself was written in C, and a lot of the low-level hackery needed to be done in C.


There was, though, a massive amount of stuff that didn't need to be done in C. If we'd had a high-level language ready to go nearer to the beginning then we'd have had a much better handle on the compiler interface, a place for people to play much sooner, an easier way to write tests, and a good leg up on the standard library.


The gee-shucks architect: Good and bad


One of the things I decided early on was to downplay my own skills.


Let's be blunt and up front here. I'm good at what I do, and I do a lot of things that a lot of people can't manage. Parrot's first mark and sweep garbage collector was put together in about four hours while I was sitting in my local public library (no wifi but comfy chairs) and it worked pretty well. The hardest part about it was doing the dull stuff involved with it. I'm reasonably certain that's not normal. And no, I don't know how to say "Yeah, I'm good at doing that stuff that makes your head explode -- it's easy" without coming across as an arrogant prat, so I don't. (Say it. I've no idea about the arrogant prat bit)


On the other hand, I'm usually really uncomfortable being up front about what I'm good at (much to a number of people's deep annoyance), and I really, really didn't want people afraid to touch code I'd written because they thought it required some sort of skill at brain surgery to alter.


One of the running jokes was that we'd know Parrot was ready for release 1.0 because all the code I'd written and checked in had been rewritten. I cultivated this, to the point of occasionally checking in bad (working, but crappily written) code on purpose, partially to give the code gnomes something to gnaw on and partly to enhance my rep as an adequate but not great coder. If people thought the code was deeply magic they wouldn't want to touch, and I knew we needed people not afraid to touch.


This was partly a good thing. We got a lot of people who hacked in on parts of an interpreter system that they might not have otherwise dared touch -- if I could deal with it and I was just OK as coder then they could dig in. We got a number of people who might not otherwise get involved actually get involved because of this, and that was a good thing.


Unfortunately the downside there is that I lost a lot of respect from some people, because I was viewed as, at best, an adequate coder. When you've got people who's sole measure of personal worth is the code they produce, well... you can see the problem.


Overcommitting the architect: Bad


I had a full-time job (with more than full-time time commitments), a family, a marriage that's been rocky for ages, and Parrot on my plate of things to do. Any one of those things was enough by itself, and two would keep anyone busy. I had all four, and they all suffered in one form or another. (Parrot, unfortunately, didn't always get the short end of the stick either) Bluntly I had more things to do than any one person could reasonably do, and I didn't have the sense to back out of some of my commitments until things got past bad for me.


Running a project like Parrot, where the scale's damn big, requires a minimum amount of attention, and for a while I didn't have that attention to give it, and what attention I did have was mostly wasted fighting with Leo.


Professional courtesy: Good


One of the things I insisted on was that people behave professionally. (Not, mind, as adults -- I know plenty of adults who behave badly, and some of the kids (and yes, Brent, you and Zach counted at the time :P) involved with parrot behaved better than some of the older adults) I wasn't looking for the sort of cult of personality that pervades the perl, python, and ruby development camps -- I find that rather distasteful, both as the potential target and as a general participant. I also remembered the perl5-porters mailing list when it was more a shark tank than a development mailing list, and I did not want that.


All-volunteer or not, I viewed parrot as a professional engineering project, and I wanted people to behave accordingly. For the most part everyone did, and I was happy about that.


Not telling people to shut the fsck up and cope: Bad


Professional courtesy is fine and all, but if you're running a project, with volunteer labor or not, sometimes you need to tell people to suck it up and deal. More importantly, there's nothing wrong with a judicious exercise of authority, especially if the exercise of authority is part of the job you hold. I was the architect for the project and it was my responsibility to design the system and get that design implemented. Part of that meant that yes, I should have told people to cope. Shying away from confrontation doesn't do anyone any good -- I did, and the project suffered for it.


People not shutting up and dealing: Bad


This one was, to some extent, out of my control, but it was still a problem, especially with Leo. If you're going to take part in a project, that's fine. If a feature or something is under development and you want to chip in, that's fine too. When the decision is made, though, shut up and cope if you disagree. If it's not your call, then once the call's made you deal with it. Odds are the person who made it also has some issues with it, but there are reasons for it to have been made, reasons you may be unaware of, or not understand. Regardless, projects don't go anywhere if decisions keep getting rehashed over and over again.


This was in part because of me not putting my foot down enough, but even in the cases where I did it was often ignored, and that was a problem. If you don't like the decisions being made in a project you register your objections when appropriate, and if things don't work the way you think they should you go away and find something else to do. The world's a big place with a lot of things going on, and nobody in a volunteer project needs to deal with you bitching about or subverting decisions you don't like. Register your complaint and either cope or go away.


Deferring to contributors: Good and bad


Parrot was all-volunteer, and because of that I didn't push people much, and took what I could get. It made sense, since how could I reasonably put any pressure on people who were donating their time and efforts? Well, I should have, at least more than I did.


When you're running a volunteer project, there's nothing wrong with asking people to do things, and expecting that they'll do them if they say they will. People are sensible and know what they can do, and will either step up or not. While you can't demand people do something they've not volunteered for, if they say they will, then you have every right to expect they will do it, and it's fine to ask people to do something as long as you can take no for an answer. Being volunteer does mean people may bail on you with little or no notice, and it means that you take second (or third) place behind other activities, so you need to keep on top of who's doing what, but there's nothing wrong with asking.


I did my best to respect people's other commitments -- everyone's got a life outside Parrot and honestly if the choice was someone doing parrot and making a mess of their home life or bailing on us, I'd be happy to forcibly kick people out to go deal with the important stuff. Not that I had to, but Parrot was just software and on the whole software's just not that important, not in the grand scheme of things.


Keeping mum about Perl 6: Bad


Mmmm, perl 6, the original reason Parrot started, and after a couple of years a nearly irrelevant thing in parrot development. I hear it's more important now, which is fine.


I'll be blunt. I don't give a damn about perl 6 at this point. Haven't for years. I'm not a big OO guy (when we started I had a knee-jerk dislike of objects, a problem I've since shed, though as a performance guy I think they're overused) and perl 6 was getting deeply OO during the design. Plus dealing with Larry as a designer was... well, it was a pain in the ass. I finally gave up in disgust on perl 6 when we lost over a week of Larry's thinking about perl time to the 5 disc DVD releases of one of the Lord of the Rings movies. (The first one, I think, though it's been years)


There was also a lot of creative tension at times in the whole design process, and I hold the dubious honor of being the only person I know of, outside his kids, to get Larry mad. (And yes, he does get mad, though in a nice way. Go figure) This is one of those "laws and sausages" things for most people. Does it matter to you how much waffling Larry's done, or how many times (and over what) he and I just-barely-didn't-shout at each other, or how many things I flat-out told him I wasn't going to implement if he designed them in, or implement even if he forbade it, or how bloody long some things took? No, it really doesn't. What ultimately matters is the final result, the rest is just development crap, and no different than you see in a lot of other projects.


Not, mind, that I think Perl 6 is going to be bad. I don't. I respect Larry immensely as a designer -- he's good, and on some days he hits great, and I say that even disagreeing with some of his decisions. There are a number of good people working to make the design happen, too.


I just don't give a damn, but I kept mostly silent about it, and I think that, for me, that was a mistake. Others may disagree, which is fine, but the perl 6 development process has not been trouble-free, and I think it could stand to have a lot more light shined on it. Looks like that's happening more now, which is a good thing. It just should've happened earlier.


Leo: Bad


Normally I wouldn't name names -- it's unprofessional and a bit unbecoming. Unfortunately you'd have to be completely uninvolved with parrot or desperately clueless to not know how well we did, or rather didn't, get along. Leo was the single biggest mistake I made with the project.


Leo, bluntly, is a massive pain in the ass, and because of him Parrot was about a year behind where it could've been when I left. I spent far more of what energy I had dealing with him than anything else, rehashing old settled design decisions over and over again, putting up with snide comments, shots on the design (often complaining about missing design documents that actually existed), and whines about the way things were designed and how badly they were. Something, bluntly, I found infuriating since most of the things he complained about were for languages that he didn't program in. This would include perl, python, and ruby. Leo, as someone who never used any of those languages, knew better than those of us who've worked on the cores of one or more of them what was good and bad.


Let's be clear. Yes, Leo writes a lot of code. Yes, he goes and implements features. Those are good things under normal circumstances. Unfortunately his code's difficult to get into, not all that great, and puts off anyone who wants to modify it, pretty much leaving any system he's worked on impenetrable to most anyone else. The features he does implement fresh didn't follow the documentation for those features, and he would re-implement the same system over and over again rather than working on something new. His interpersonal skills drove a lot of people off the project as well -- I'm far from the first who left Parrot because of Leo. Bluntly, Leo was far more of a hindrance than a help, and I put him in the position to kick the crap out of Parrot, so I've nobody to blame but myself here.


Yes, I know, volunteer project, and the third (or is it fourth) rewrite he did on the garbage collection system's fast! Woo! But... so what? In the mean time dealing with him wasted so much time and energy that exceptions, IO, and events never got dealt with, a dozen or more good people went away permanently, a large chunk of parrot's a bloody mess, and a lot of it is prematurely optimized into near-unfixability.


I shouldn't have made Leo pumpking when I knew he'd driven off other developers, and I should've told him to go away the first time I ignored my perl6-internals mail for a week because I couldn't deal with him. I didn't, and Parrot suffered.


The wrapup



So, what did I learn from all this?


  1. Don't confuse motion for progress.

  2. Don't undermine your own authority.

  3. If you have authority, don't shy away from using it where appropriate.

  4. If you expect professional behaviour you'll get it, so expect it all around.

  5. The easier you make it to do things the more people will be able to do

  6. Be open and above-board always. If you're embarrassed to do so because of what's going on, that's a sign of a big problem.

  7. It's only software and not worth your life.

  8. People suck. Deal. (And never forget that you're people too)


I expect there's more, but there you go, and take it for what you will.

Posted by Dan at October 20, 2005 05:16 PM | TrackBack (0)
Comments

Alright, finally the dirt! :)

But seriously, the second paragraph under "Leo" ends in mid-sentence.

...

Weird, the URL text field complained about my URL (craptaculus . com) because it has u s . c o m in it. The regexp must be a little greedy.

Posted by: Tom at October 20, 2005 07:08 PM

I took us.com out of the blacklist -- most of the blacklist entries lack \b prefixes. Gotta fix that at some point.

The missing paragraph's now not missing too. Thanks for the catch.

Posted by: Dan at October 20, 2005 07:29 PM

Y'know, I figured Leo would make an appearance; I just didn't think you'd use the exact same term I use for him.

Thanks for your work on Parrot, even if you're done with it now. If nothing else, I learned a lot, and not just about programming.

Posted by: Brent Royal-Gordon at October 20, 2005 08:55 PM

Wow! I have not followed Parrot closely for a long time, so you leaving was a mystery to me.

But how you describe Leo reminds me of one particular person in the early days of perl5-porters.

Posted by: Graham Barr at October 20, 2005 10:12 PM

I recognize that your analysis is about the development of parrot in particular, but honestly i'm more interested in perl 6, and I think it's interesting that you weren't.

My personal feeling about parrot from day one has been that the emphasis on serve a niche which only theoretically exists (a VM that can run various dynamic languages) instead of focusing on perl 6 in particular, from the top (language) down, was the wrong move.

I've certainly watched and even supported parrot's development, but nothing I have seen to date has really changed my world.

When pugs came onto the scene, which did take exactly that approach, I feel that much much much more progress was made towards there being an actual perl 6 some day than had been made to date.

As someone who makes their living with perl, a language that is increasingly falling behind other mainstream languages, I really do want progress. The details of the VM are totally not interesting to me, as long as it works, and at acceptable speed.

Parrot's being so disconnected from perl 6 was never good for perl 6, in my opinion.

Of course this is all just opinion, and i'm a fairly disconnected observer, but again, as a fairly disconnected observer, nothing as made me as excited about the potential for perl 6 as seeing the two projects, parrot and pugs, working to meet in the middle somewhere. That's how it should have been from day one.

Still watching,

--Josh

Posted by: Josh Wilmes at October 20, 2005 10:24 PM

Well, you might think that. That's fine. Languages don't run on magic faerie dust -- something has to drive them, and the fact that you don't care is irrelevant. That something has to be there, it has to be maintainable, and your ignorance about the magic doesn't change that fact. Hell, the fact that you can reasonably be ignorant about it is a testament to the people who've busted their asses to make it so you didn't have to care. (And in the case of perl 5, I was one of those someones, though a minor one in the grand scheme of things)

Let's be blunt. I'd have loved to have language and software start together from the beginning. They didn't. They couldn't. Not from day 1. Or 10. Or 100. Or even 1000. THERE WAS NO PARTIAL PERL 6 SPEC TO IMPLEMENT THEN. By the time Larry finally got enough to go on, I'd long since handed the responsibility to get perl 6 the language implemented on to someone else. And unless I misremember... you weren't involved. If you're unhappy about that, go bitch at Patrick Michaud, he's the one I handed that responsibility off to.

If we'd waited for Larry to get something implementable before implementation started, I'd probably have had a much more pleasant life. And perl 6 would be dead, because waiting three or four years for something to go on would've killed it. Nobody would've cared.

You've not been involved. That explains your ignorance, and there's nothing wrong with that -- we're all ignorant about a lot of things. This would, however, be a good time to recognize that ignorance and perhaps not render an opinion on that which you have insufficient information to make a reasonable judgement.

Posted by: Dan at October 20, 2005 11:05 PM

Interesting that you chose to interpret my statements that way. My point was not "VMs Are Irrelevant" (While an argument could certainly be made that perl 6 could target whatever VM happened to be handy, I am not making that argument), but that, in my opinion, if you want to accomplish perl 6, you have to focus on perl 6.

And yes, I was also disappointed that the specification of the language was developed in the way that it was. However, I do think there was also a lack of emphasis on implementing what *was* available as soon as possible. Instead we had dozens of broken toy languages and a vm that wasn't clearly connected to the overall goal of being perl 6 some day. It was a thing of its own, and (again, opinion only) this wasn't a good plan.

As far as my involvement goes, i'm not sure that i'd say that I haven't been involved. There certainly were points in time when I was making a lot of patches, though I did consciously choose to not get involved in implementation, just enforcing coding standards and the like. I did not have the time or (frankly) the brain to get too deep into parrot innards, so I always picked around the outside.

And, as progress seemed to slow, my interest did fade, and I did stop actively participating. This is a pretty typical pattern in open source projects- momentum is a hard thing to maintain, and the appearance of squabbling can sap what motivation remains.

I haven't stopped following the discussions though, as I have since day one.

--Josh

Posted by: Josh Wilmes at October 20, 2005 11:17 PM

Well, I owe Josh a bit of an apology -- he has been involved with parrot, something that's faded out of my memory with time and uninvolvement. So... good for you, Josh.

On the other hand, having been involved you should've known better what was involved getting something like perl 6 going, and the timeline. This whole thing started summer 2000. That's almost five and a half years ago. With a very large chunk of that operating with no perl 6 spec to speak of to go on. So... bad for you, Josh. You were inside enough that you should've known better.

Posted by: Dan at October 20, 2005 11:22 PM

From my point of view, and I’m speaking as a complete outsider, it looks as though Perl6 would have kept floundering forever, were it not for one Autrijus Tang who started actually implementing the language (if at first only as an experiment) – and in the process, forced to the surface a lot of hidden self-contradictions and ambiguities in Larry’s designs. The Perl6 project only gained momentum, it seems to me, once implementation and design started to iterate in tandem.

So for all I can tell: code trumps prose. Well, that’s hardly news.

Of course, what I can’t make any call about is whether an effort like Autrijus’ could have started any sooner. But if it could have, it was a huge mistake (on whoseever part it might have been, if anyone’s in specific) not to do so any earlier. And if it couldn’t, then having a design process which did not enable it was the huge mistake. (And I think the latter is what Josh is talking about.)

So how much of what I wrote is nonsense?

Posted by: Aristotle Pagaltzis at October 21, 2005 02:06 AM

Aristotle - While it's true that Autrijus more or less jump started the mostly dead P6 process, the difference is that Autrijus is a brilliant guy. Shit code is shit code, whether or not everyone holds hands and imagines it's not. Even so, Pugs remains a prototype, whether or not P6 will come to pass and if it will not be obsolete and irrelevant by the time it does, remains to be seen. While the folks more resistant to reality tend to deny that, in spite of the popular mantra amongst the perl community these days that if people tire of the abuse or get used up that there will always be more fodder for the grist mill, it's the individuals who matter, who make a difference and, while Dan is putting a bit of a soft spin on the events and the impact to parrot in the long run, those who know compilers, those who have been around for 20 years or more in the field understand that there is a big difference between getting something done and getting something done right.

And Dan, while it's a little late to dump what so many have been saying quietly in dark corners, I'm glad you grew the balls to pipe up and put it where its at. ( http://www.amk.ca/conceit/parrot.html has been around for a while but, like so many who are lone voices, people can pretend that problems don't exist.) The number of people telling me, telling everyone else that Leo is horribly unfit for the technical lead and that parrot will suffer massively for it is growing.

The only thing the meek inherit is their ass being handed to them.

Posted by: hfb at October 21, 2005 07:06 AM

Dan, did the technical reaction to your sandbox plans also play a role in your departure?

Posted by: curious at October 21, 2005 08:22 AM

"The gee-shucks architect" - OMFG I am a genius - I hate to brag but Christ I am smart! I am also really modest. So modest I would occasionally write bad code so as not to overwhelm others with my sheer brilliance - genius, no? By doing this I was able to entice programmers who aren't super-geniuses like me into waters above their heads, helping drag the whole project down.

It is no wonder Perl6/Parrot has gotten absolutely nowhere in 5-6 YEARS with people like you running the show. Thank the gods for Python, Ruby and mono.

Posted by: perl5-programmer at October 21, 2005 08:25 AM

I don't remember any reactions as such to the sandbox plans, so no, they had no role in my departure.

I handed off the designer hat to Chip when I finally wised up to the fact that I was desperately over-committed, and Parrot was the least important thing I had on my plate.

I bailed on Parrot entirely when it became clear that Leo was pushing to get a lot of his bad ideas into the core and had a good chance to get a bunch of 'em in. That was just pissing me off, and there wasn't any resaon to hang around to see it, so I didn't.

I've no idea what, if anything, will ever happen with the sandboxing stuff, amongst other things.

Posted by: Dan at October 21, 2005 11:17 AM

What if he was right? What if he really was that smart? Wouldnt that make all of this really sad, especially cosidering what it could have become?

IMHO: Anyone who dares to say someting like that about himself so publically cannot be the sort of person who doesnt love his work deeply.

Posted by: Non_Perl_Programmer at October 21, 2005 11:19 AM

How reasonable do you now feel is the goal of producing a multi-paradigm language? In particular, I suspect the semantics of basic data elements in the supported languages must match for the system to have a chance of performing efficiently. A simple example is strings: they are either immutable or not, and the engineering trade-offs for many implementation decisions will depend on knowing which decision is made. Is this, in your experience true, or do you find it doesn't really matter that much?

Posted by: Scott David Daniels at October 21, 2005 12:52 PM

perl5-programmer: Very smart programmers who are effective in leadership positions are frequently self-deprecating. No one likes an asshole, even if the asshole is correct.

But there's a difference, as Dan has demonstrated, between self-deprecating and self-immolating.

Posted by: Ruby_Programmer at October 21, 2005 01:10 PM

It's great that you got this out here. I hope you can maybe just do the Parrot thing better on your own in the form of Cola - I think the dynamic languages could well use some consolidation in the form of a shared VM.

Me, I don't like Perl, so I was always more interested in the VM part then in the Perl part.

Posted by: Manuzhai at October 21, 2005 02:50 PM

I stumbled accross this site when I was trying to figure out what the heck is MMD (good post, btw), so what I'm about to say may identify me as very clueless, and I definitely want to say this in the nicest possible way.

From the post:

/* People not shutting up and dealing: Bad


This one was, to some extent, out of my control, but it was still a problem. ... If you're going to take part in a project, that's fine. If a feature or something is under development and you want to chip in, that's fine too. When the decision is made, though, shut up and cope if you disagree.

... [much later]:

Does it matter to you how much waffling Larry's done, or how many times (and over what) he and I just-barely-didn't-shout at each other, or how many things I flat-out told him I wasn't going to implement if he designed them in, or implement even if he forbade it, or how bloody long some things took?
*/

Posted by: Max Lybbert at October 21, 2005 02:53 PM

I think the goal of a multi-paradigm language is either reasonably obtainable or completely laughable, depending on the paradigms you're looking to merge.

If you consider object orientation a paradigm (I don't -- at this point I look at it as the next step in procedural programming the same way that non-goto control structures and functions are steps in procedural programming) then yeah, no problem. You can mix OO code in with non-OO procedural code. You could, I suppose, with sufficient drugs, mix it in with functional programming if you wanted.

I don't think it's possible to get a language that really mixes procedural, functional, and logic programming (or any two of the three) in a way that works. Bridging between them is a different matter, and that's both possible and interesting, but a single language that crosses real paradigms? Nah, I don't think so.

This is all semi-independent from some of the other sorts of language considerations, stuff like the mutability of things (objects, strings, numbers, whatever) and suchlike stuff. Those considerations can make life interesting, certainly, though in nearly all cases they're irrelevant to the language itself, and only really become an issue when you're playing with implementation details. (Not all of them, certainly, but a lot of them)

Posted by: Dan at October 21, 2005 02:59 PM

Yep, a reasonable catch Max, though in this case it turns out not to be the case. Our not-quite-fights took place while features and language issues were being discussed, proposed, and otherwise beaten about, not after they were set and done. That is, my "no way in hell" declarations to Larry were when he was putting up some "I want to do X" proposals. Once they hit fixed-in-stone state I always made sure they'd work, no matter how much I disliked them.

The sole exception were rules forbidding behaviour in the interpreter engine. Those I'll freely admit I ignored and left them to the language compiler (which I'd passed on to someone else to handle) to forbid. If someone wanted to bypass it and get access to raw parrot functionality I was fine with that.

Posted by: Dan at October 21, 2005 03:11 PM

I do have to admit that I did leave out the line "*If* *it's* *not* *your* *call*, then once the call's made you deal with it," and that there was a possibility that the subjects of the Larry interactions had been things under your call. That would appear to cover behavior in the interpreter engine.

But, as you said, these "not-quite-fights" took place before the call was made.

Posted by: Max Lybbert at October 21, 2005 04:20 PM

Dan,
Thanks for posting this. I've followed the development of Parrot from a distance since the 'Pie-thon' challenge. I was really hoping that you would succeed as I think that an open, performant and portable VM with good support for dynamic language features would be a huge boon for everyone.

The lessons learned are valuable and reinforce some of the things I've learned in my career.

I would love to see more post-mortems like this for other projects. My experience is that these are some of the best insights of the issues involved in 'real-world' development.

Posted by: Ken R at October 21, 2005 06:17 PM

It's not that I doubt that this Dan guy is a genius. In fact, I am sure that he is. But being a genius and being an effective leader are not the same thing. And checking in sub-par code so you don't blow everyone away with your smarts is *not* self deprecating - it's blowing smoke up your own ass. Do you think Torvalds, Rossum, ESR, Icaza or other open source super-stars pull that sort of thing?

All of this ultimately has to rest at Larry Wall's feet though. He is the one who abdicated his leadership role and gave these guys way too much control over the future of Perl. Do you think anyone would get away with the shit Dan pulled on Larry with Linus?

Posted by: perl5-programmer at October 22, 2005 01:40 PM

I'm always amazed at how rude and obnoxious some people are on the Internet. Ah, well...

Dan, thanks for all of your informative write-ups, and for the work you did on Parrot. (I followed Parrot for a long time, but between my job, family, and school work, I never had a chance to participate.) Keep us posted with respect to Cola developments, and good luck on the home front.

Posted by: Sam at October 22, 2005 11:17 PM

I wholeheartedly agree with the last post.

And pretty much everything Dan posted in his post-mortem stinks like the honest truth to me.

I have been quietly watching the parrot saga on a daily basis through the medium of cryptic daily emails, for 3 (or 4?) years now, desperate to know w.t.f. was really going on. I was constantly on-the-verge-of starting to write a compiler and attack the docs - but something would happen on the list to put me off - usually a rude/patronising email. Parrot seemed a scary, badly documented, world to get involved with.

However, for every question I ever asked, Dan gave a long, thorough highly-technical yet easily understandable reply. Others were not so polite.

I had suspected that Dan was underplaying his coding skills as he would regularily check-in huge complex new peices of functionality. It is true that there were often jokes about the quality of Dan's code though. I couldn't understand this. His code worked. His designs were great. Surely that's what is needed in working towards v1.0?

I also couldn't understand why, on many-many occasions, when Dan said - "No, we're not touching this until we're functionaly complete" - certain people went ahead and rewrote code that was working (probably plenty-fast-enough efficiency). And why every time we got some decent docs, someone would change the inteface and invalidate them. I spent £20.95 ($37) on a nearly completely useless book. By the time I had finished reading it - it was pretty-much all out of date. I'm sure that put a few other people off, too.

But, the revolution will be properly documented. Marx new this, Mao new this, the PHP creators knew this... and that's why we got what we got. ;-)

Why do open-source *programming* projects need these Christ-like figures? And why, when a serious open-source project gets a leader who says that debate is OK do certain people jump in and nail them up? Shunning publicity and not shouting me!-me!-me! does not make you a bad programmer. Not having a website with a photograpoh of yourself in a tuxedo and dark glasses does not make you a bad programmer. Not saying something controversial to get yourself on Slashdot every week does not make you a bad programmer. Building a geeky cult around yourself is *not* necessary to write a good computer program. I repeat.. *computer program*..

I for one Wholeheartedly support Dan Sugalski and wish him the best of everything in his future projects.

(How about a small, simple, fast, portable, vitual machine that has easy-to-use hooks for everybody to hang whataver nasty code they want onto the skeleton? I for one would take part - assuming documentation & 'actually shipping' was seen as more important than premature optimization. Just start with the docs and don't nail yourself to a cross this time ;-) )

Sorry for the rant. I feel better now.

Posted by: basil_programmer at October 24, 2005 05:33 AM

Thank you for posting this post-mortem. Although I don't follow Parrot development closely, and certainly haven't contributed anything (if only I was independently wealthy and didn't need a day-job), I've always found Parrot more interesting than Perl6 - and more important as well. Being able to easily combine modules/libraries written in different languages would be, IMO, a greater boost to Perl than Perl6.

I do hope Parrot doesn't die, and hopefully, people do consider your writings and do something good with it.

And I sure hope you aren't (permanently) lost to the perl world.

Posted by: Abigail at October 24, 2005 06:54 AM

[OT] Seems an international problem, these ex-parrots ;-)

Posted by: Paul M at October 24, 2005 07:05 AM

My unsolictied, unwanted advice:

Marriage is more important than any OSS project, even if it is named after a most majestic bird (well, ok, I just like Monty Python). It is more important than any programming project or job. Reconcile or split. Heal yourself or free yourself (and free her). Marriage fulfilled only in its legal sense is not good for anyone involved, and it is especially damaging to children. (Think of the children! No, really.)

Points 1, 7, & 8 are fabulous. I would only ammend #8 as:

People suck. But they are worth it. (And never forget that you're people too. And you are worth it.)

- A divorced programmer from a family that should've divorced (since they wouldn't reconcile) 8 years before they did.

Posted by: Anonymous at October 24, 2005 11:18 AM

I hate me-too postings, but here goes:

Thanks Dan. Thanks for the work and for the post mortem after the work.

Sometimes honesty hurts the messenger as much as the recipient, and I'm glad you came clean on the whole thing.

Posted by: clintp at October 24, 2005 12:23 PM

Very interesting, though not exactly reassuring. But, see point 7 :)

Posted by: Baczek at October 24, 2005 04:23 PM

Parrot was definitely scary and badly documented, and even when I did have the time for it, it was just too big a mess for me to try to untangle and actually make anything more than an infintesimal contribution. Of course, VMs are complicated scary things, but inaccurate documentation is even worse than none at all, because you never know what's right and what's not... you don't know what you don't know.
The thing that really put me off parrot, and why I stopped following the project, is the fact that it hasn't really made much progress feature-wise. It's a neat idea, but I really really would have liked have things like events and async I/O and exceptions and all that other fun stuff in there. Heck, I might have even used it then, but "then" is over 2 years ago now. I still look forward to having a cross-platform, cross-language VM to play with... but I'm not sure parrot will be it.

Posted by: crzwdjk at October 24, 2005 04:46 PM

I can't say that coming clean was particularly easy (and as Elaine said I soft spun on some of the things that went wrong) but getting it out was good. Hell, just cutting and running was good.

Getting things sorted out at home's a bit more interesting than might otherwise be for other people, but that's the way it goes, and it is definitely more important than Parrot was, though it took me far too long to realize that.

And next time... next time I spend a lot more time writing docs, and a lot more time not letting code into the repository that doesn't follow the docs.

Posted by: Dan at October 24, 2005 05:03 PM

I feel like I should say something, because I was there, knew there was a problem a long time before Dan left, and had the authority to kick one (or both) of them off the project and chose not to use it. Here's the shitty part about being a leader: sometimes you send your team through hell with the full knowledge that you're doing it, and the full knowledge that it's the only way.

Dan and Leo are both good developers, the best Parrot had at the time. They have very different technical perspectives, but those differences complement each other nicely. I didn't really want Dan or Leo without the balancing influence of the other. (Chip is doing well in the balancing role now, but he only became available a couple months before Dan left.) Dan and Leo also have very different personalities, but not so different as to make working together impossible (just difficult).

To those who could learn a lesson from the Dan/Leo conflict: if you ever find yourself in a similar situation (no matter which side you're on), apologize for your behavior and make friends with the person you keep butting heads against. I've had to do this a few times in my life, and as painful as it is, it's yielded some of the best friends and co-workers I've ever had.

The other subjects are familiar to anyone who's ever worked in open source. Writing documentation is boring and it's difficult to get volunteers to do it. People who create programming languages are just humans like the rest of us; we love them as brothers and help them as friends. Various people get disillusioned with the goal on the way to completing it; new people come to fill their shoes. There's much work to be done in Parrot and I'm glad to be part of it.

Dan, thanks for walking many miles with us. I'm glad I had the opportunity to work with you. I'm sorry for the parts that were hell, and I hope that in a few years you'll look back and also remember the parts that were fantastic. Grief heals with time (this I know from much experience).

Posted by: Allison at October 24, 2005 07:14 PM

Thank you, Dan.

Posted by: Autrijus at October 25, 2005 09:14 AM

Great write up. Just curious, if much of the problems in the Parrot project are due to conflicting engineers, instead of leaving, why don't you branch the project and be more selective about contributors?

Posted by: Damien Katz at October 25, 2005 09:39 PM

Well... I did fork off parrot. Haven't done much with it, since I've been more in recovery and catch-up mode than anything else -- things had gone about as far as they could possibly go and gotten about as bad as they could reasonably get, so I wasn't up for much of anything for a while.

I'm not dead, yet, though. I'm actually feeling better. :)

Posted by: Dan at October 25, 2005 10:26 PM

Hmm. I haven't been involved with anything Perl6 related for a few years, at this point... I inserted myself with full force into an effort to produce open source documentation for the Perl6 language as it was being hashed out, but that mailing-list based effort died -- in that I found it, at the time, to be almost impossible to make progress at a sustainable rate, the O'Reilly books would always (by design) be far ahead of any online open-source efforts, and there were very few people around who were interested enough or fool enough to help take over the process of trying to wrangle documentation into existence in an effort to push the Apocalypse/Exegesis efforts into more formal and fully-explored descriptions.

So it's sort of interesting to me to hear some of the things that I was most worried about two years ago -- that the Parrot team was threatening to hit a gigantic wall waiting for the most primitive possible decisions of the language to be locked down and fully explored -- explored from a completely different angle, and by someone who -- unlike my own trifling and minor role -- was a very central part of the efforts. I vaguely recall from that time that the central effort, in absence of more concrete documentation, boiled down to creating a haphazard suite of tests which would act as ersatz specs for the more primitive or unspecified language behaviors. It's certainly not the way I would have approached it, but hell, if it turned out to work, that's great, and if it didn't, then, well, we all just gotta go with what we got the best we can.

I did and do have high hopes for the language still, however. Programming is for me no longer even the slightest bit interesting -- it is simply a chore, and a tedious one at that, re-inventing the same basic algorithms over and over because of what-should-be-pointless minor differences. Honestly, I haven't "invented" any substantially new code for ten years -- merely rewritten the same basic cornerstones for half a dozen different companies. In short, I wanted Perl6 to be a language in which large-scale code reuse was actually possible, _really_ possible on the scales that commercial software projects increasingly demand, and I saw the distinct glimmers of things in Perl6 that would/will make that happen.

Because frankly, we don't have that yet, tiresome apostles of OO and functional programming and insert-language-name-here and everything else be damned. Couple that with the tug of compsci purists as language designers who insist that the only good code is code that is impenetrable to the loin's share of the actual coders on the planet -- thus rendering libraries or entire languages largely pointless for anyone but the categorically and tediously brilliant -- and the corresponding tug of the many, many coders I have worked with over the years who render code impenetrable through coding that is simply spectacularly and repetitiously bad, and I've certainly... hoped... for a minor miracle to occur in the discipline. I don't think any language can, in and of itself, fulfill that, but I certainly am enamored of the possibilities engendered by any language that attempts to _try_.

And, as I said, there are glimmers of something truly great, in the underlying Perl6 combination of concepts. So I have to say, I never felt the slightest bit of irk at the various fluctuations of the language -- even where something happened that I strongly disagreed with, as an outside observer, in nearly every case the end result was better over time, and never worse, simply because the various wishlists and features were being molded into something that tied various disciplines together in a way I had/have hope for. I had/have a lot of confidence in Larry as a designer of something that will, overall, be fantastic to actually _use_, and that hasn't changed.

But I did quickly become exasperated at how difficult it was to pin down basic functional conundrums, like the transformations between primitive types or the basic behaviors of operators -- especially where there seemed to be an inconsistency.

I think the opacity of the design team in many of these decisions was a big mistake. Specifically, it was not merely impossible to know what a particular design decision would turn out to be, it was impossible to have any clue as to _when_ it would be decided, or _if_ it would be decided, or even what order the various design issues were being tackled in, or whether or not a particular issue was being worked on at all, or whether the design team had even _spoken_ to each other during a particular span of months. Having patience with design decisions is one thing; not having the slightest idea if that patience was warranted, or if it would ever be rewarded, was a bit harder to deal with. I think a lot of potential volunteers were turned off by what _seemed_, in public, to be organizational disarray on the part of the organizers. And those of us whose talents ranged more in the narrowly organizational side of things, I think, were stuck in a position of volunteering to be essentially useless/unwanted baggage, or not contributing to the effort at all.

I think whatever failures exist in the Parrot and Perl6 efforts, back in those days, were attributable to brilliant computer scientists who had neither the skill nor desire to shepherd the efforts from a managerial or organizational effort.

Has that changed, in the time since? I don't have a clue. I notice that there's more going on at present than just the Apo/Exe pairings, so clearly, it is at least marginally different in the intervening year or two.

The lessons I tend to draw might be entirely dissimilar to yours: I don't know enough about the politics of the core team to know, since I was never privy to any of it. But I think that while the design of the language was, indeed, very well served by having hundreds of individuals provide input, with a strong central designer to sort through the wheat and chaff, both the design and the implementation of any project of this scale is increasingly impossible to do part-time, as an unpaid position.

I can only imagine how things might have been different, back in the beginning, if the Perl6/Parrot effort was a corporate, university, or otherwise sponsored R&D effort -- an effort that could afford to pay people to dedicate their time, and which therefore could lock in the needed talent as a matter of employment, and not of charity.

This is long, and terribly disjointed, but that's the probable result of me being so bothered by it since I myself moved on, out of necessity, from my own volunteer role. Thanks for the public debriefing Dan... I can't say that it makes me feel better about anything, but it certainly provides more data points for my own thoughts on programming in general.

I still believe Perl6 will be resoundingly successful -- just not for a wickedly long span of time.

Posted by: Mike Lazzaro at October 25, 2005 10:53 PM

"I feel like I should say something, because I was there, knew there was a problem a long time before Dan left, and had the authority to kick one (or both) of them off the project and chose not to use it. Here's the shitty part about being a leader: sometimes you send your team through hell with the full knowledge that you're doing it, and the full knowledge that it's the only way."

Allison, being an appeaser is not being a leader. You went to the Larry school of leadership and it shows. Years of choosing to let shit happen while sidestepping it yourself and acting like you 'led' the way through it instead of just avoided it and waited to see what happened isn't leadership. And this is why the community is where it is now.

I'm sorry, years of bitter frustration watching those who should lead say that they were leading but really just stepping back and pulling a papal wave and smile should end.

Perl remains in need of strong leadership that's willing to make unpleasant decisions as this is an unpleasant community at times.

Posted by: hfb at October 26, 2005 01:59 AM

Human beings are unpleasant sometimes. That's just a fact of life. Choosing to emphasize and encourage the good in life isn't a weakness. In fact, it's much harder than emphasizing the bad.

Second lesson: when you find yourself in a situation where you can't reconcile with the other person, no matter how hard you try, you can either spend a great deal of time countering their head-butts or ignore them and keep working.

Today is my day this week for working on Parrot. The topic at hand: tree transformations (a critical component in the compiler tools suite). Yay! :)

Posted by: Allison at October 26, 2005 10:47 AM

Allison, it seems to me that there are two kinds of people in a position to give leadership advice: those that realize that they've made mistakes (like Dan), and those that have been successful. With respect to Parrot, you don't appear to fit in either of those two categories.

Honestly, what isn't wrong with the Parrot project at this point? It's taken so long that it's lost its credibility; Leo has been running people off the project for years, and there's no reason to think that will change; and it's lost its designer. Can you let yourself see that? Here's a lesson for you: facing reality is the first step toward changing it.

Posted by: Sam at October 26, 2005 11:48 AM

It's not Leo that's been putting me off patching anything in the parrot tree: Allison has mentioned above the “compiler tools suite”, which apparantly consists of PGE, L::AG, this tree transformation thing, etc, but I have no idea how to contribute to this! The versions in the repository of all these things are never the latest.

Patrick Michaud seems to do all his work away in some darkened basement and only lets everyone else see what's going on when it's finished. I can't contribute because he's already rewritten half the code while you're waiting for a new release, and I'm put off going near it because I just have no idea what's there.

The same goes for anything else in this ‘compiler tools suite’; not that I know what this is of course, there's no roadmap in the repository, and no code either.

So, I've love to contribute folks, but I can't, and I don't particulary want to be in the dark all the time.

Posted by: Anonymous at October 26, 2005 05:11 PM

Sam: There are problems (every project has some), but they're also solvable. My response is to get rid of all my management hats and spend my time on code. I just started taking one full day a week for Parrot. It's amazing how much more productive a full day is than the same amount of time spread out over the week.

Anonymous: Given a choice between coding, and talking about the code I'm writing, I'll choose coding any day. :( I'll try to do better. Here's a start.

Posted by: Allison at October 31, 2005 04:09 PM

"For those of you who choose to continue, I wish you all the best of luck.", Jamie Zawinski, 31-Mar-1999

Thanks Dan.

Posted by: mj41 at November 4, 2005 08:40 AM

Thanks, Dan, both for all your good work on Parrot and for your thoughtful write-up. I particularly appreciated your engineering approach to documenting a clean design with replaceable subsystems. Your insistence on professionalism throughout was also exactly what the community needed.

Thank you again, and good luck in your future endeavors.

Posted by: Andy Dougherty at November 7, 2005 10:10 AM

Thanks dan ,both for all your good work on Parrot and for your thoughtful write-up.

wait for cola project!

Posted by: parrotsqlite at November 8, 2005 11:25 AM

Probably no one's reading this anymore -- ah, the internet, what a pile of ephemeral trash -- but I just found it, so I'd like to make a few comments. First, thank you Dan for piloting Parrot as long as you did, because it was no doubt a frustrating job. However, in my experience your leadership contributed to the impression that Parrot and Perl 6 were guided by capricious and absolute decisions. One week feature X would be done in way Y, but sometimes the next week it would be Z, with no knowing when a decree would be revised. You consistently gave the impression that your decisions were arbitrary and justified solely by your will, even though they may have had sound technical justification, and this robbed them of legitimacy.

Second, to those who think an early Perl 6 would have helped: I tried, and while I'm not nearly as smart as Autrijus, I don't think even he could have constructed anything decent on top of Parrot. Much of the work in producing what exists in languages/perl6 was spent finding and working around design and implementation bugs in parrot, and adjusting to arbitrarily-changing Parrot Design Decrees. Only by abandoning parrot, I believe, was Autrijus able to get as far as he has in implementing Perl 6, and only by continuing to abandon it will he ever finish. Look at his latest efforts -- he's clearly ensuring that Pugs will run in final form on something other than parrot, probably because he realizes that parrot is an impossible architecture for which to write code.

Finally, the named criticism of Leo is rude and unproductive. I agree that he's opinionated, forceful, and sometimes hard to work with, and have sufficient experience to make this judgment -- remember, he started contributing to languages/perl6 before parrot. However, singling him out for public criticism is nothing more than vindictiveness and pettiness. Dan: you lost, you left, move on.

/s

Posted by: sean at December 27, 2005 04:28 PM

parrot and perl6 have never attracted a large and diverse enough team, that continues to be a major issue. because the team was not large and diverse, there are no mechanisms to load-balance good contributors or filter out bad contributors. looking at the perl6/parrot blogs and summaries on the web, i see a team that revolves around fewer than ten key contributors, with the real inertia behind maybe three individuals, and i don't count larry wall as one of these three. larry underestimated his own presence in the project when he declared perl6 to be a "commmunity" rewrite. his own MIA status early on was very problematic, although i understand it was partially related to his health.

what happened to the rest of the perl community? many of the individuals who have written perl books in the last three years appear to have no presence or even interest in the parrot/perl6 project. not that you can blame them...the projects were kicked off in 2001 and by 2003 there was nothing actionable. compare this to the mono project, which in the same period has distributed an initial release, bugfixes, a major second release, and many apps. mono perhaps offers a lesson on the value of some corporate involvement: resources, dedicated paid development time, support, and also an expectation of real progress. since larry branched off from o'reilly, the perl community seems to have lost even that.

at this point so much of the perl community has lost interest that the parrot/perl6 projects essentially are over. the few contributors making glacial progress with microscopic release and throw-away implementations (pugs) seem to very much not care about delivering.

Posted by: grumpY! at December 29, 2005 01:23 PM

The comparison to the Mono project was a common one, but one that was always deeply flawed. Mono definitely benefitted from corporate help, but it was Microsoft's corporate help they benefitted from far more than Novell's.

The Mono project had, for all intents and purposes, a complete language specification in C#, a complete VM specification in the .NET engine (which may well have been a proposed ECMA spec by the time Mono started, though if not it was still near-completely spec'd out), a pretty complete library specification, a working reference implementation, and a working development environment.

This is not to denigrate what the mono folks did. There's a lot of work there, and they're definitely to be commended for what they've done. They just started their project on top of a foundation that a couple of dozen man-years of work by very smart people built.

That's probably the biggest difference in perceptions between the projects. Parrot and perl 6 (the two projects are, and have been, separate for years) were doing in public that which Microsoft did in private, only without the resources that Microsoft has to muster for projects like this.

Don't, though, go writing Parrot off because it's taken ages. The Mozilla project took ages, as did Apache 2. Granted, so has Duke Nukem Forever, but the point is that large projects take a long time, and it's awfully easy to write something off because it's run past the average programmer's (very short) attention span.

Posted by: Dan at December 29, 2005 03:38 PM