Rice flour actually works pretty well in things you'd use wheat flour for but don't need gluten (so good for cakes and pies, bad for bread and cream puffs) but the stuff you can get at the market is damned gritty. (Check the ethnic food section -- my Stop 'n Shop has it for about half the price of the big box 'organic' food places) I've been told you can pick up food mills that'll get it as fine as wheat flour, but I don't use it nearly enough to warrant the expense, so I'm stuck with the rough stuff.

All flours you use in cakes work by hydrating, or soaking up water. How fast flour hydrates is largely function of the grain and the grind. Finely ground flour soaks in water faster than a rough grind, and different grains (wheat, rice, sorghum, or whatever) soak at different rates as well.

Rice flour is perfectly happy to soak up the water and get nice and smooth but the rough grind means it takes ages for the water to penetrate to the center of each piece and soften it up. Definitely much longer than you'll spend mixing and cooking a standard wheat flour recipe (it takes more water, too) so if you don't plan for that you end up with a gritty cake. Not horrible, but not great, and certainly not a texture I'd use for a birthday cake if I could avoid it. With some extra time and a blender, I did.

Grit-free, gluten free chocolate cake

2C Rice flour
1/2 C Sorghum Flour
1/2 C Potato starch
3 oz Chocolate
1C Cocoa
2C Sugar
1t salt
4t baking soda
1 1/2t xanthan gum
2/3 C flavorless oil
2T Vinegar (optional, but not bad)
1T Vanilla
2 eggs
2C water

Get the water warm but not hot to the touch -- a couple of minutes in the microwave is fine. (You're not cooking the rice, just taking advantage of the fact that warm water works better than cool for hydrating) Stir in the rice flour and the sorghum. It'll make a bit of a paste, but that's fine. Make sure all the flour's wet -- you don't want pockets of dry flour. Cover and let stand at room temperature for an hour, at least. (Longer is fine) If you want to stir it every half hour or so that's fine but not necessary.

After your hour's up, heat the oven to 350F. (And check! A cheap oven thermometer's probably cheaper than a bag of the rice flour, certainly a damn sight cheaper than the xanthan gum. Use it, because your oven dial lies!)

Line two 9" round cake pans with parchment paper. (No other pan treatment is needed, and don't grease and flour anything -- just the parchment, and it really doesn't even need to be all that exact a fit)

Dump the rice and sorghum mix into your blender. Blend (lid on, since cleaning the ceiling's a pain) for a minute or two. Add in the eggs, vanilla, vinegar, oil, and salt. Blend at least until mixed. Your batter will still be a little grainy, but that's okay, it'll cook out. Blend longer if you feel like it, the extra air helps make the cake fluffier.

Break the chocolate up into smallish pieces and melt. (probably a minute or two in the microwave -- do it in 20 second chunks, stirring between each nuking) The chocolate doesn't have to be hot, just liquid, so don't go overboard.

With the blender running, add in the chocolate. (That's what the little hole in the center that the cap's in is for, and why the cap comes out) Whiz it up until it's blended.

Now, if you've got a heavy duty blender (I don't) you can add the rest of the ingredients and whiz together in the blender. Otherwise dump everything into a mixing bowl and mix.

The batter will be a bit rubbery, so plop it into the pans and spread until everything's basically level.

Bake for 30 minutes, or until a toothpick stuck in the center comes out clean. (If 30 minutes isn't enough then check every 5 minutes until they're done) Let 'em cool, then de-pan, frost, and eat.

It's also nearly white, something cow butter from the supermarket isn't. Close enough that I think you could reasonably do white buttercream. (Which I don't, as it means either lots of titanium dioxide or crisco. Either way, ewww)

8 oz goat butter
4C powdered sugar
3 oz chocolate, melted and cooled
1T vanilla
3-4T liquid (water or milk of some sort)

Melt the chocolate and let it cool. It should be liquid, but only barely warm. (Otherwise it'll melt the butter when you add it, which is no good) I find a minute or two in the microwave, in 20 second intervals, works pretty well.

Beat the butter until it's light and fluffy. Beat in the powdered sugar a cup at a time. (Or more, if you like the whole 'exploding cloud' effect) Beat in the vanilla, and the melted chocolate.

The frosting'll be a bit grainy, so add in the liquid a tablespoon at a time until the consistency is what you want it to be. Water works fine, as does milk from any handy mammal. Soy 'milk' or coconut milk work too, though I find they've each got a distinctive flavor they add to the frosting, so check to see if they match what you want.

This is enough frosting for the outside of a 9" cake. If you've got a 10" cake, or want to do between the layers of the cake with it (Why? Use a good jam instead. Raspberry, strawberry, and cranberry all work, and it's easy to cook up a quick fresh batch) then increase the amounts by 50%.

For folks following, I wouldn't expect updates with much frequency (past the next couple of posts) nor much computer talk. I do enough of that at work; it'll probably mostly be food and the occasional snarky grumble.

For a configure system we probably don't care much about what type an individual data element is. int, float, boolean, string, whatever -- ultimately these things will get turned to strings and splatted into templates, so adopting a dynamic-language-style morph-on-demand basic data type is a reasonable thing. So we'll do that.

There are things we do care about aside from the actual value of a data element, things we use to decide what rules need firing off. For example, we care if the element:

1. Has been initialized
2. Has changed
3. Has a default value

All of these things are needed to help decide which rules to fire off.

#1 is straightforward enough. If a data element has no value, then we can't use it yet. That means when we're ordering rules to see which we're going to run, we need to run the ones that set the value before we run the ones that use it.

#2 is also straightforward. If a rule depends on X, and X changes, then we probably need to re-run the rule. The exception here being that we don't need to run the rule if the rule doesn't actually do anything we need. For example, if rule Foo depends on X and provides Y, and we don't need Y, then there's no point in firing the rule off. Assuming we don't allow side-effects. (Which we aren't, but I'll get to that another time) Whether a data element has changed can also be an interesting question -- who decides? Does the solver engine decide, by comparing the old and new values? Or does a rule affirmatively declare that it's returning X and yes, in fact, X should be considered changed? (And what about the cases where it returns X, with a changed value, but says it hasn't changed? We'll get to that another time too)

#3 is somewhat less straightforward. Configure systems generally don't start from nothing when they run -- there's seed data that has adequate but not necessarily optimal values that you can prime the system with. As a for example, you may not know what the best C integer data type to use is, but it's certainly safe to default to "int". The system may probe for a better value, perhaps long or int64 or something, but in its probing it can use int until it finds a better answer. Or you may, by default, assume you can't fork (a safe assumption) but you can always come back and probe to see if you can.

The utility here is that you can ship safe defaults, enough to get the system in a state to go find better answers. Sure, maybe you've filled in the defaults with lowest-common-denominator answers from the C89 and ANSI operating system standards, but that's enough to get you in a position to figure out what the actual useful answers are.

Alternately you could mark values loaded up from the default cache as changed, but... maybe not. There are actually cases where you don't want to mark a default value as changed.

Consider, for example, the person who's actually packaging up things for distribution. We've kind of hand-waved up until now on how these 'trivial' shell scripts, batch files, or default config.h (or whatever) headers and such get created, but this is a good time to un-wave the hands.

The packager could run the build on each OS you want to support and collect up all the results. But... ewww. That means the packager needs to be minimally fluent with each system, and have access to each system, and if you lose access (or lose the person with fluency) you have a system fall off the list. Not good.

On the other hand, the packager has a perfectly fine configuration system handy. If there's also a set of default values for each system (which someone else can build, can be handed down from maintainer to maintainer, or set to something really simple) then the packaging can just load up the defaults, generate files, and then go on to the next system. If you treat the defaults as unchanged then, if your rules are correct, you won't need to actually run any probes and you're good.

When the end building person gets the package they run the generated simple shell scripts which builds the configurator with the defaults, which then loads in the default values, starts probing, and rebuilds itself. Same templates, same rules, the only difference is that the packager doesn't do the probe and rebuild step, since they don't have to. (Or, rather, if they do it's a sign there's a missing default)

None of this is all that interesting, but it's handy to have as a base when looking at the more interesting bits. Which we'll do later.

You'll want it for cases where you have a template rule that needs to have a separate instantiation for each file in a directory. You'll have it because when you're designing your rule language you're going to want run a rule for each element in a list or array which means you really need separate instantiations of the rule for each element. You'll have it because it's a shiny thing and programmers just can't resist Teh Shiny. (Admit it, you know that's true)

There are four different ways to handle this.

You could consider each rule as a closure, close over the environment, which includes the current array entry or filename or whatever.

You could have a way to clone an existing rule and make changes at clone time, basically a rule factory.

You could consider embedding the configuration system's compiler in the configuration engine and just treat the code as text, and recompile things on the fly, like perl's string eval.

You could consider a partial compilation system with placeholders, and basically implement Lisp macros to dynamically generate rules on the fly.

Each of the options has its own strengths and weaknesses.

The closure option is the least attractive of the three options. What you're generally closing over -- what the rule depends on, or produces, or the files a template operates on -- is really more metadata than data. And yes, you can close over that, but it feels kind of awkward. Which isn't to say that closing over the environment is a bad thing, even in a functional language with no side effects (which I hadn't mentioned, but there you go), but it's really not the right thing here. The only real upside is that this should work, but since it's hacking an existing concept in a way that's not particularly true to the concept

The rule factory's more interesting. It's a very limited sort of thing, specialized to do one thing and presumably do it reasonably well. That's not bad, if what you want is what it does, and in this case it is. I'm not all that fond of limited solutions like this, though, as they often feel like someone just stopped thinking and made the best of what they had thought about.

The compiler option's a common one, and it definitely has its advantages. The big downside is it means that new rules mean doing text mangling and then compiling the result -- we've got to have the compiler handy, which makes things more complex. Plus this kind of text filtering can be problematic, as we've seen in perl. (Though as we've established, one thing that a configuration processor is going to have to be good at is template processin) On the other hand, having the compiler handy means that you can provide the rule system as text so if something goes wrong the clever end user can fiddle with it to make it right, without penalizing the end user who doesn't want to be clever.

Then there's the lisp macro system, though it's mis-named, or at least deceptive these days since most people think of macros in terms of text, or recorded little action things in a spreadsheet. Basically you manipulate the compiled code, copying, adding and potentially removing chunks of executable as you string it together into a final form. Very much like the rule factory, actually, except without stopping thinking too soon. (The rule factory is, in fact, a subset of the macro system) Creating a new rule is just a matter of copying the code for an old rule and making some judicious substitutions.

Personally, I find I like macro system form best. It doesn't have to be completely exposed in the actual configuration language, merely implemented that way. (Much the same way as compiling Fortran into object code that does all its control flow via CPS -- the fact that the underlying implementation involves continuations is irrelevant (kinda) to the compiled language, which doesn't. It just means you can potentially do Clever Things with less pain)

It also just feels like the right thing to do. If the config system is potentially going to be dynamically generating dozens, or hundreds, of rules on the fly and pitching them at the solver engine, something a bit less haphazard than text substitution and recompilation seems in order.

They're one of those things most people never think about, or if they do it's at best as a user. You probably ran the autoconf-generated shell script, or did whatever else the package you installed needed. It did some strange magic, built some files, and you were done. Easy, right?

Well, okay, maybe not so much. If you've ever poked around inside the guts of an autoconf-generated script you're probably aware there's more concentrated evil in there than you'll find outside the source for Clippy. It's just nasty.

That's not really a surprise; it is a multi-unix shell script designed to probe the environment of a system it knows so little about it can't even really count on a properly functioning shell. And yeah, I know, it's a lot better than it was, but still... Ewww. Just because the insides are justified (and possibly obligatory) doesn't make it any less Lovecraftian. (So I'm not knocking autoconf, it just scares me. A lot)

As a developer, configuration systems are a pain, for a few reasons. The first is portability -- by their very nature, configuration systems go look for things that aren't on the system you're developing on. (Because if they were you wouldn't need to go look to see what they are) Writing portable code is annoying, because it requires making good assumptions from the beginning, and that's tough. Especially because, if you're at all comfortable with the system you're working with, it means making assumptions that are different than your default ones.

The second big pain with config systems is that it's a hassle to track down bugs, again because it involves systems different than your own. How can you tell that the code you built on your 32-bit x86 linux system doesn't work when configured and run on a 64 bit AIX box with more than 4G of memory if you don't actually have a 64 bit AIX box with more than 4G of memory?

And the big final annoying thing about configuration systems. Like test frameworks, they're probably completely different from the app you're actually writing. That is, there's likely little or no overlap between the mindset needed to do thing you like to do (writing the game or chat client or whatever it is) and the thing you need to do (write the configuration system). Unless you're writing a configuration system, in which case it gets all meta.

Then there's the whole not knowing what you actually need to go look for. Most of the annoyingly quirky things (like the PDP-11's wacky 32-bit integer format, which is neither big nor little endian, but rather middle endian) you had to deal with in the past have thankfully died enough that you don't have to care, but if you've never had to deal with the vagaries of how shared objects are built and work on a half dozen systems you'd never even think to go probe for them.

All of which is why people generally reach for autoconf, and you really can't blame them. But autoconf is evil, and for a complex system it's inadequate as well, since there's a damn sight more than just a makefile and config.h that you need to build up when you're configuring.

So anyway, configuration systems. What the heck are they, anyway?

If you think about it, there are four bits to configuration systems:

1. Rules
2. Probes
3. Template instantiation
4. Seed data

Seed data gives you sane, functional (though possibly only barely functional) defaults. Probes gather information from the environment to override the defaults, rules decide what probes need to be triggered and what values are produced, and instantiated templates are what you end up with. Possibly in an iterative way, since a lot of environment probing involves instantiating templates (that is, little C programs) which are compiled and possibly run.

It's all very dependency based; to instantiate a template you need to have all the input values, which different rules produce, and some template instantiations depend on other instantiations, which depend on yet other instantiations, and so on. Not all that much different than what make does, only with built-in actions (really built in, into the executable, rather than predefined ways to invoke programs) and generic dependencies rather than time-based file dependencies.

Or, if you like, the mutant bastard child of Prolog and Template::Toolkit.

More interestingly, if you're going to build a configuration system, you can actually manage to do so from scratch, with a bare-bones shell script or batch file, a C compiler, and the linker, which is a pleasant change from the past, where you couldn't. What I mean by this is that, if you assume a C89 standard C compiler (which, given that it's been 18 years since that standard was made seems safe), you can manage to get everything you need to probe the environment.

Think about it. What do you need to build our configuration system? You need to read in and parse rules data and templates, you need to do dependency ordering, you need to instantiate templates, you need to spawn off subprocesses to check individual bits and pieces, and you need to read in the results of those probes. That is, you need fopen, fread, fwrite, fclose, system, and a boatload of templates. Everything else is built into the configuration progam.

All of those things are guaranteed in C89. (And yeah, I know, there may be systems that don't handle them all right, but at this point it's pretty safe to assume they all do) Putting the code together so it compiles and links with close to no knowledge of the system is pretty straightforward too. A series of cc commands followed by ld for unix and its like, for Windows you use whatever the most common compiler is, or you use some environment variable to do it (and have a prompt at the beginning of the batch file for the C compiler and linker -- you can be pretty sure that if someone's running a .bat file it's windows, after all)

Yes, that does mean that maybe you're not using the compiler and linker options the user really wants, but that's fine, because once you've built yourself you just go ask and do it again. Or provide a way to set some environment variables or command line parameters if that's your preference. Worst case there is you need to compile everything twice, but you're probably going to do that anyway, since a full-fledged configuration engine needs to be able to do system-specific stuff, which means probes and rebuilding. The good bit about all that is that you don't need anything to configure your configuration system. Which is good, because otherwise you end up chasing your own tail, and that's annoying. (not to mention hell on your back)

And, of course, once the engine is already built, anything else that uses the engine for configuration already has a prebuilt set of values that can be pretty easily filled in to skip most of the probes. (Or you can package the config system with whatever needs it and just build it, since a half dozen or so extra compiles and links aren't going to noticeably increase the length of time it takes to probe the system for settings)

There's more, but this is enough for now.

(This is all Jim Keenan's fault, I should mention that, in case anyone's curious)

Anyway, I've been working on a GOL for Tornado. It works too, which is cool, and it's a little surreal to do a full generation with no explicit loops in it anywhere. And, as expected, it's shaken out a number of bugs in Tornado, both in the ops that're used by the program itself, but also in the error handling (which is, alas, pretty close to nonexistent at the moment), the assembler, and most importantly the garbage collector. Or something the GC tickles, at least, as memory usage grows until there isn't any more, and performance grinds to a halt.

It may sound a little strange, but I like programs that kill my VMs, especially if they're small and viciously abusive. Those are the best kind, as they wrap up all sorts of nasty edge cases and stressors in one handy package. Makes debugging easier, makes testing easier, and shakes out lots of bugs in one go so you don't have to spend all that much time writing test cases, which leaves more time for debugging the code.

There's a second handiness here, in that life, if you look at it right, is a vector program. (Hence the being able to write it with no loops) If you play it with a torroidal board, as is traditional, and you have a vector rotation op (which may not be traditional, but is pretty darned simple), it's just a series of vector rotates, adds, and compares. Turns out that a generation takes 34 Tornado ops to run, regardless of the board size. And yeah, 34 lines of code for life isn't all that great, but as assembly goes, not at all bad, really.

This sorted out all sorts of GC issues, some minor flow control issues, an off-by-one error in the assembler, and a variety of bits'n'pieces.

The runtime isn't too bad either, though I can't say as I'm thrilled with it. On my spiffy new MacBook Pro (2GHz Core 2 Duo processor), for 500,000 generations, I get:

10x10: 12 sec
10x20: 16 sec
10x40: 28 sec
10x80: 50 sec

which, honestly, isn't that swell, though you can see that dealing with larger quantities of data scales pretty nicely -- the 12 seconds for a 10x10 board was pretty bad, honestly, but the steps as the board gets larger make it clear that there's a lot of overhead in the 500K generations, but the overhead doesn't grow linearly with the amount of data.

Still, you might think that 10K generations/sec on a 10x80 board still isn't that great on a 2 GHz processor, and you'd be right. It isn't, and I don't much like it. (Bleah!)

Luckily for me, there's a rabbit and hat handy. Those numbers are with GCC with no optimizations on. With -O3, the time for a 10x80 board drops to 17 seconds, which is a more respectable 29K generations. Still not as nice as I'd like, but definitely better. (This is the place where I question the algorithm I use, as well as considering Duff's Devicing the loops in the vector ops, but I'm putting that off for now, because I want to write a templating preprocessor to do that, so I can do 'em all in one go)

I should be cutting a 0.06 release in the next day or two, time permitting.

Update: Now that I go do the math, a 10x80 board has 800 cells, and at 29k gen/sec (okay, 29411.76 gen/sec) that's 23M cells/sec considered. Or about 85 cycles/cell to do the figuring on this beastie. Okay, maybe it's not all that bad, unless I screwed up my numbers.

There's one big limitation, which I'm unhappy with. For reasons that annoy me, when waiting on a handle for results, only callbacks posted in the function the handle represents will be processed. I'd hoped to have it so that callbacks from any function in that particular instance of Tornado would be processed, but I wasn't thinking clearly and put together a design that required waiting on multiple condition variables simultaneously, which just doesn't work with pthreads.

Of course, now after spending a few hours hacking that out so that each handle was independent, I realized I could have them all coordinate on a single instance-wide condition variable without much trouble. I'd rejected it for performance reasons -- it'd mean all the waiting threads would wake up and have to check to see if their individual wait condition was satisfied -- but on further reflection that was kind of silly, as there's unlikely to be many host threads waiting on the instance mutex at any one time, so the CPU time spent waking them all up and having them go back to sleep just isn't likely to be a big deal.

Anyway, this works, which is cool. Next step is either to get the Windows build working (I need to get Win-specific thread stuff in) or to get the threads-spawning-threads and messaging code in. Not sure which. I think I ought to get win-threads working first, but message-passing and thread-spawning is a lot more interesting.

In the mean time, I get to leverage perl's wad 'o platform-specific knowledge and just bodge all the objects together into whatever perl needs, and if someone twiddles with a ruby or python version they can do the same thing.

I'd feel guilty, except the module library still weighs in at under 140K. I think I can live with that, 'specially as the library built for Devel::Size is 57K and there's not nearly so much code in it.

Windows still doesn't quite build right, but if all goes well it will in the next day or so. Assuming you've got the same development environment I do (which'd be activestate perl and the .NET 2003 dev kit), and if you don't then I probably ought to make PPMs to go with the source tarball. 'Cause, what the heck, having the perl module makes testing this stuff a lot easier. Yay for dynamism and all.

(No, there's no particularly interesting insights here. This is all grunt work, though I am so looking forward to learning Windows threading model enough to get its thread stuff into Tornado...)

Unfortunately there's still some manual bits, like you have to copy the tornado shared library into the lib/ directory in the perl module's build directory when you're building and testing, but that's only because I've forgotten the Weird Magic needed to get MakeMaker to add in library paths and so I cheat. It's probably dead-simple, but for now this is easy enough, and I can write test driver programs in perl rather than having to abuse the main.c in the main tornado source directory. (Which is terribly inflexible)

That does work out, in its own way. Use the flexible language to do flexible things, the fast (I hope) engine to do the fast things, and let the glue code sort out the interfacing. The way things really ought to be, I think.

I did this in tornado with the opcodes, in large part because I fully planned on having a simple oploop to start with, and more complex oploops later. I also knew I was going to need to yank out opcode info for the assembler and whatnot, so pre-processing the source made a lot of sense. (This was something that we did quite successfully with Parrot)

I didn't do it with the various MMD math ops. This is something I'm coming to regret.

Firstly, there's just a whole wad of cut'n'paste code in because of it, and that's always really error prone. Yech for errors. Fixing bugs in the boilerplate is always a pain when it's boiled over, and you know that you always find the bug in the code that's been copied everywhere but miss one or two spots in the cut'n'paste fix.

Second it really sucks when you realize that you want to roll out a functionality change, but you've got a half-zillion places you have to wedge it in because it's in the boilerplate.

Feh. Should'a done it in the first place. Now I have to go back and rip out the guts of math.c and replace it with code that handles shorter vectors on the RHS of an op for looping. And possibly the LHS too, I'm not sure. (There is the question of what you should do -- loop back to the beginning, or pad with zeroes, or just run the length of the shorter vector)

Ah, well, it's only code. No reason to not rip it out and make it better if there's an actual win to doing so, and it'll make the source code smaller too. Smaller is usually more comprehensible, and I like comprehensible.

(I really ought to cut an 0.04 release -- fixed a bug on the train and added in vector/scalar versions of the math ops. Maybe tomorrow)

Anyway, it's in, and as a bonus there are some (very) minimal docs in the docs/ directory that I've cleaned up a little, the README file is mildly useful, and the garbage collector won't collect up and kill global variables. I regenerated the ops table too, so opcodes will have constant numbers. Not a huge deal, but stability of opcode numbers is nice. I'll probably toss and redo the table and bump the version number before release, since there's a lot of crud in there that I think I'll redo when I have time. (Like there are _vv _ss and _sv versions of opcodes, which made sense until I built the MMD system into tornado, and are now mostly just annoying fiddly things)

I probably ought to dig out some subversion plugin or other that'll automatically generate a changes file.

New version at http://www.sidhe.org/~dan/things/tornado-0.03.tar.gz. Feel free to have at it, and I think the comments on the blog are even fixed. (A mixed blessing, since that means blogspam again)

(And if you snagged the tarball before 10:30PM Eastern (I think we're GMT -500) re-snag -- it now builds on linux, and I tossed the damn redundant ops and went full MMD for dispatch. We have it, so we might as well use it, and I can add in static analysis later if I want)

It only builds as packaged on OS X, and requires perl (but, then, if you're on OS X you have perl) and the docs are abysmal to non-existant, but if you want to play it's at http://www.sidhe.org/~dan/things/tornado-0.01.tar.gz. It's under the BSD license, so it's pretty much a go for it, good luck sort of deal.

It ought to work on Linux if someone kicks the config system a little. I'm working next on getting it to build on Windows, and that should be interesting only because the threading's so different there.

Don't expect much threading, though. This is, right now, entirely vector oriented, with the sole exception that calling into the interpreter to execute a function spawns off a thread and that function runs in the background. The cool tornado threading stuff comes next, but one thing at a time. Good chunks of vector functionality are still missing, and It's also not too friendly to program in either, but that's OK too. It does enough for me to hang more stuff onto it, and I'm fine with that.

If you play with it, expect to make friends with gdb, though. :)

And before anyone decides to mention it, yes, it does have a big function pointer table and do indirect function calls for opcode dispatch. There's enough indirection in place to fix that later, but opcode dispatch time is, bluntly, noise for Tornado right now if it's operating on any real data. Adding together two 100K element vectors (which can be done with a single op) eats more time in memory wait time than all the opcode dispatch overhead in your entire program.

This may change as the threading ops get added in and I start doing more work with threads rather than vector data, but then I go and massage the opcode function code generator to emit a switch statement or computed goto core or something. Or bite the bullet and write a JIT. Something like that.

Anyway, in my Copious Free Time (which is to say barely ever) I've been abusing Tornado, trying to get the basics of the thing to work. And... now it does. Woo! I'll grant, that's the sort of Woo! that comes from spending an order of magnitude more time on a dead-simple problem than the problem actually warrants (which means, of course, I was doing threads work) but I don't care. Woo! it is.

Okay, so Tornado doesn't as yet have garbage collection, and while it has most of the vector ops in there's no internal threading yet (merely external threading; Sending a function request to an instantiated Tornado instance spawns a new thread to run it), and some of the error checking's a bit dodgy. Which means it leaks like a sieve and falls over if you stare at it funny, but it works anyway. Besides, adding a GC's only an hour or two of work, so that's no big deal.

I've gotta admit, it's kind of cool to write a program that does fairly complex work on megabytes of data, when that program's all of twenty or thirty likes of assembly code and has no explicit loops in it.

I think this beastie's nearly ready for a release for folks to fiddle with, if they're so inclined. Or not, that's OK too.

One thing I have noticed that's really odd is that printf and its ilk don't work from inside Tornado on my OS X box. This is not a huge problem (I've made friends with GDB) but it is puzzling. They work fine from the driver program but not inside the engine code that lives in the .dylib. I have no idea why, and in practice it doesn't matter (since the engine can't do any IO itself by definition) but it's weird. I expect there's some odd flag or library I need to link in, or I should've waved the chicken widdershins around my laptop. Something technical like that.

The nice thing is that there's more than just a skeleton in place. Now there's a good chunk of meat on the bones, and a framework that makes it straightforward and pretty easy (courtesy of some of the lessons I learned from getting Parrot up and going) to extend things. This makes me... happy.

Of course, I expect getting this running on Win32 will make me Unhappy again, but that's OK, I can live with that. The code I need's in example form inside the perl 5 sources (and yeah, I know, that is a scary thought). Just gotta handle thread creation and joining, mutex creation, locking, unlocking, and condition waiting/signalling/broadcasting. (Which, I realize, is done a little differently under windows, but I've already abstracted it away a lot, so I'm fine there, I think)

That's all great, since it helps provide you with a safe engine, and since we're really shooting to be able to run untrusted code that we get from someplace we don't entirely trust. Agents and calculation engines and macro programs and whatnot, and it's nice to have some assurance you're not going to have your system compromised because you downloaded and installed the WhizBang Dive Control plugin for the Aerial Deuces flight simulator.

Of course, the bad part about making it impossible to talk to the outside world is that you can't talk to the outside world. Which is, sometimes, a problem. Sometimes getting data at function start and returning data at function exit isn't enough -- you really want to call out into your environment or send back intermediate results. And while that's possible to do this way, you have to have the environment essentially send in seeker threads looking for data to return. (Which is actually kind of a cool concept, if you think about it. Damned awkward to actually write code for, though)

To get around this, Tornado has a mechanism where the environment can register callback functions that threads can invoke to do things. What the callbacks do is up to the environment, so it's a good idea to make sure that your Tornado programs match up with what your environment exports, but since we tag things that need verification we can at least check function signatures for callbacks. This way the code can have some access, and the host environment can regulate that access and enforce any sort of policy it might want. (Rate limiting, destination filtering, or whatever) When in doubt, punt, right?

This does have some issues, of course. The host does have to register and manage things for the engine, but something's gotta do that, so too bad there, especially since in the simple case it doesn't have to actually do much, just proxy the request. That's the simple case.

The complex case is when the host program actually needs to do some work to satisfy the request, and the things that makes that difficult are, of course, threads. Tornado code probably can't just invoke the callback function -- Tornado's heavily threaded and most host programs... aren't. I fully expect one of the common cases is for a single-threaded main program to embed a Tornado engine to do its thing, and even in a multithreaded case you don't generally want to have random threads calling into yourself. (Yes, I know, if you're embedding a threaded engine and providing callbacks you should be careful. But you should floss after every meal and exercise regularly too, and we all know how often people do that)

For that, then, Tornado allows you to note that a callback isn't thread-safe. If you do, Tornado gets mildly clever.

While it's not made clear to the calling program, each Tornado engine instance has a master thread attached to it. When you start an instance the thread is created, and when you make a call to have the engine do something the call's made to that thread, which then dispatches the call. When any thread running under that engine makes a callback, the callback request is routed to that master thread, which then makes the call into the embedding program but... only when the embedding program has said it's ready for the engine to execute its callbacks.

Basically the embedding program calls a Tornado function that drains the queue of waiting callback functions, so those functions get called from within the context of whatever host thread is draining the queue. (This way the embedding program can do the things that have to be done in a specific thread, like GUI or database calls, from the right thread)

Alternately the embedding program can note that a callback is always safe (because, for example, the embedding program is multithreaded and has properly protected the code), in which case Tornado code will just call them from whatever thread in the engine that happens to need to make it.

How the embedding program waits, and how it manages callbacks, is a matter for another post.