Moonbase

Just a quick note that I’ll be making an appearance on tomorrow night’s Linux Link Tech Show at 8:30PM Eastern Time.

I’ve noticed lately that some web application frameworks seem to encourage the practice of automatically putting database transactions around view/controller execution. This may seem like a convenience, but it encourages precisely the wrong way of approaching transactions— at least if your application ever does anything else besides interact with the database. Yes, you’ll want to use transactions when your views or controllers are using the database, but in some cases having a transaction over the entirety of view/controller processing is far too coarse-grained.

See, there are two basic principles to bear in mind when working with a transactional store:

1. Minimize transaction size and length.
2. Avoid performing non-transactional IO during a transaction.

Minimizing transaction length not only saves resources, it also improves throughput and reduces the likelihood of creating conflicts with other transactions.

However, you do also need to be prepared for the case where conflicts occur. I see an awful lot of code out there which assums that transactions always succeed, and this is dreadfully wrong—if there is writing going on, you basically always need to be prepared for the possibility of a transaction being aborted due to a conflict.

In many cases the appropriate way to recover from a conflict is to retry the aborted transaction after a short delay; repeated conflicts are typically typically with some kind of exponential backoff and a bit of randomness introduced to keep retrying transactions from butting heads too much.

And retrying is one place where the second rule comes in: if you did a whole bunch of non-repeatable IO during the course of the transaction, then you’ve pretty much just screwed yourself.

I’ll give an example of what I saw yesterday which got me thinking about writing this post:

A web application was set up to automatically put a database transaction around each request. One of the functions of this web application is to accept browser file uploads, updating some bits in the database before and after the upload takes place. A consequence of this is that the application would have a transaction held open for the entire duration of the upload—which could be minutes or even hours. This ties up resources associated with the transaction, creates the potential for other transactions to block (to the extent that pessimistic concurrency control is used by the database), and lastly creates a great deal of potential for update conflicts to occur.

And at that point, what happens if the transaction does fail? Since an upload transaction included reading the entire file from the browser, we can’t easily retry it: the browser is only going to send the file once.

The correct solution in such a case is to use multiple, finer-grained transactions during the course of processing the request. Those transactions would be shorter-lived than a mega-transaction spanning the entirety of view/controller processing, are less likely to provoke update conflicts, and can be individually retried when conflicts do occur.

So I’ve been getting mail to the effect that Ruby 1.8’s concurrency primitives are still all screwed up even as recently as 1.8.6p111, and that fastthread is consequently required to get things to work reliably.

I find that really frustrating, as I was hoping that once fastthread was merged with 1.8 as the new thread.rb, I could get out of the fastthread business altogether and rely on the 1.8 maintainers to deal with it, but unfortunately things don’t seem to have worked out that way.

I think what I may do this point is re-enable fastthread for all 1.8 versions, and hope that Ruby 1.8, as a Ruby implementation, goes away sooner rather than later.

I’ve been plugging it elsewhere, but I’m really enjoying multi-instrumentalist Sophie Madeline’s album LOVE.LIFE.UKULELE, and I figured I should give it a plug here. It’s the sort of quirky, poignant thing that I really get into. You can listen online at the above link, and bandcamp also has an option to purchase high-quality downloads at a price of your choosing.

Also, here’s an interview with Sophie, complete with bonus song. And her single The Stars.

fastthread has pretty much outlived its usefulness at this point; it is no longer really necessary on the most recent versions of Ruby 1.8, and things depending on fastthread is actually a hinderance on JRuby and Ruby 1.9 and so forth. A number of gracious people have contributed patches to make fastthread installable as a noop package on those platforms, and before that I actually prepared fastthread 1.0.3 last June, but it was never packaged.

Worse, Mongrel’s SVN also appears to have lost the revisions I originally pushed to it (which I know I did, because the revisions had git-svn revnos). So, I’ve put fastthread up on github and have released 1.0.3 myself.

I’ve been away from blogging for a while due to various life complications and other commitments, but I’m back now, and I’ve got quite a bit to write about.

Watch this space.

For you.

BAMAN PIDERMAN

Interview with Dali

(Both these gems are by Alexander Butera; I hope we get to see more from him in the future!)

I don’t have a whole lot to say about science blogger PZ Myers’ public desecration of an allegedly consecrated host. I think it is fair to take him at his word that he does not seriously entertain the idea that a piece of bread could by divine fiat become the Body of Christ (which brings to mind Christ’s response the first time people started sticking nails in it ). In any case, that much is between him and God, and I generally wouldn’t expect a non-Catholic to have a Catholic perspective on the issue.

What I do hope most people would recognize is the basic immaturity of the act, even from a strictly secular point of view: in such terms, it’s akin to a grade-schooler taking from another child the special sandwich the child’s mother had prepared for him and grinding it into the dust because it was “stupid”. Of an adult academic, such an act would suggest an exceptionally poor standard of public behavior. That Myers’ act was carefully premeditated, and that the intended recipients of the Eucharist place an even higher value on it, that only makes that issue worse.

I realize that Myers had taken it upon himself to act in retaliation for threats made by a few of my co-religionists in an unrelated Florida incident. But if certain people are acting like vindictive children, an adult response does not mean similarly immature behavior, and an adult response most certainly does not mean taking it out on an entire religious group.

Update: Based on the replies I am getting, it seems that there are four issues which I need to clear up.

First, Myers (at least if his own claims are to be believed) didn’t simply obtain a wafer of the same sort of bread which is used in Catholic rites (which, absent any confusion, nobody would really care about), he specifically sought out a wafer that had actually been in the posession of the Church and had been consecrated by a Catholic priest.

Second, there isn’t a legitimate way to remove a consecrated host from the Church’s custody. The Church does not relinquish custody of a host distributed at Mass until it has actually been eaten. To this end, Catholic churches train their staff—ushers, priests, deacons, and extraordinary ministers—to observe and ensure that the consecrated hosts distributed are consumed on the spot as intended (even if, as this incident shows, they are not always adequately vigilant).

Third, even if the removal of the host in this case might have constituted petty theft (or grand theft, if the “buy it now” price on that one eBay auction is indicative of the monetary value of consecrated hosts), I’m deliberately avoiding legal considerations. From a simple, human, secular standpoint, the basic issue is that it’s inappropriate for a mature adult to take an article of extreme sentimental value to its intended recipients and destroy it in public as a deliberate act of spite.

Fourth, the original threats Myers was retaliating against were directed at a kid in Florida, not him personally. Professor Myers read about this in the media like the rest of us, and decided to take it upon himself to strike a blow for … something. (As far as I can tell, his two main accomplishments so far have been to provoke some of the same morons to start sending him threats, and to offend an awful lot of uninvolved Catholics.)

I think this article can speak for itself:

The military trainers who came to Guantánamo Bay in December 2002 based an entire interrogation class on a chart showing the effects of “coercive management techniques” for possible use on prisoners, including “sleep deprivation,” “prolonged constraint,” and “exposure.”

What the trainers did not say, and may not have known, was that their chart had been copied verbatim from a 1957 Air Force study of Chinese Communist techniques used during the Korean War to obtain confessions, many of them false, from American prisoners.

...

“What makes this document doubly stunning is that these were techniques to get false confessions,” Levin said. “People say we need intelligence, and we do. But we don’t need false intelligence.”

...

The only change made in the chart presented at Guantánamo was to drop its original title: “Communist Coercive Methods for Eliciting Individual Compliance.”

Apple’s announcement of their Grand Central technology sounds interesting, but it’s pretty light on details. Here’s my wild guess, based on the intersection of what they’ve hinted at and what I would do if I were in their shoes: my guess is that they’re integrating support for fine-grained scheduling (as in e.g. TBB) into the OS scheduler, which would enable smarter global decisions about things like work balancing and cache warmth.

Think M:N threads with better kernel support, but minus the usual attempt to make the “userspace” portion of the scheduler preemptive. If they’re smart, they’ll also include non-blocking/asynchronous equivalents of standard blocking APIs, though in principle with sufficient kernel support they could also permit tasks to use some blocking operations in more transparent ways.

I guess we’ll see how good my guess was in a year or so.