Indent four

We are running quite a number of Plone sites with the Varnish proxy server in front of them, all neatly managed by the supervisor, a process manager.

Recently some of our Varnishes unexepectly quit. Checking the supervisor logs gives an ‘expected’ shutdown state, the same message you see when you shut down a process manually with supervisorctl. But we didn’t do that.

So: A) why is varnish quitting and B) Why doesn’t supervisor restart it?

To start with (B): by default supervisor thinks exit codes 0 and 2 are ‘expected’ codes on which a process isn’t restarted. You can change this by adding

exitcodes = 0

to your supervisord.conf or adding it as an extra setting like this in your buildout if you are using collective.recipe.supervisor:

30 varnish (exitcodes=0) ${buildout:directory}/bin/varnish

Now for (A). I haven’t nailed it down exactly yet, but most buildout examples (and our production servers are still using Varnish 2.0.X, where the cli_timeout by default is 5 seconds. This is a timeout for communication between parents and worker processes in the varnish setup. I’ve found this bug ticket #329 where the cli_timeout by default has been upped to 10 seconds, because some setups might timeout when the system is under heavy load (as we found out later from our munin graphs, the systems were quite busy around the time that Varnish exitted).

I haven’t been able to find much documentation yet on exit code 2 from varnish and I’m not sure yet if if is also sometimes used for ‘graceful’ stops. Which would mean that supervisor will not let the process end when it should, but hopefully this will solve our varnish exits for now.

If you want a newer Varnish in your buildout with the default cli_timeout set to 10 seconds, try plone.recipe.varnish 1.1b1 . One warning though: On my OS X 10.6 developer machine I had to install pcre using macports before varnish would build. All my tricks to get a pcre compiled in from source in my buildout cost me 100 grey hairs and half a day of trial and error without succes. But that’s something for another blog post.

Did you run into similar problems and/or have tips or other solutions? Please let me know.

UPDATE: I’vechecked in a fix on collective.recipe.supervisor trunk, hopefully it will be picked up with release 0.12.

I’ve been tinkering a bit with process memory monitoring for applications which are started and managed with supervisord. If you want Memmon or status checking you will need the ‚superlance’ package added.

Since we are using buildout to generate all our setups I’m using collective.recipe.supervisor to generate the supervisord.conf. Adding
plugins = superlance
eventlisteners =
Memmon TICK_60 ${buildout:bin-directory}/memmon [-p instance=400MB]

Does not get the monitor running though. When starting supervisord the memmon process is exitting to soon and stopping. Running memmon on the command line with the correct options gets it running though.

After some more trial and error and searching online the culprit seems to be a recent change with recentsupervisor releases. this is what the recipe generated for the eventlistener:

[eventlistener:Memmon]
command = /Users/fred/buildouts/officious/bin/memmon -p instance=400MB
events = TICK_60
process_name=Memmon
environment=SUPERVISOR_USERNAME=myusername,
¬ SUPERVISOR_PASSWORD=mypassword,
¬ SUPERVISOR_SERVER_URL=‚http://127.0.0.1:12345

The values in the environment line should be quotes though. So changing that line to

environment=SUPERVISOR_USERNAME='myusername’,
¬ SUPERVISOR_PASSWORD='mypassword',
¬ SUPERVISOR_SERVER_URL='http://127.0.0.1:12345'

makes memmon start up correctly. I’ll see if I can get this reported/fixed in the recipe this week.(updated on trunk).

btw. (¬ ) is linebreak, but should be continued on the previous line

We are actively using Deliverance at Zest Software to theme a default Plone site  with a completely new skin. With Deliverance you run a ‘transformation’ server in front of your website like a proxy, but as an added bonus (or more as its intended purpose) you select parts of pages that the Plone site serves and insert these snippets like the content, main navigation and support menu’s  into placeholders from a static html template with css and images on the file system.

Deliverance is being experimented with and put to production use by many organisations/people already . There are quite some blog posts and notes on the web, but Documentation and howto’s are still below the level of  more mature (read: older) projects. The official deliverance 0.3 documentation gives a thorough overview of commands and transformation, but doesn’t present much real world usage or a full ‘howto’. This is not surprising since deliverance is so general in nature (fetch, transform/merge and deliver). Your individual documentation needs really depend on the sources from which you want to merge.

Installing with buildout

We are using buildout to develop all our Plone Sites. In the Plone Collective there’s an example buildout ‘deliverancedemo’ that sets up a Plone site and deliverance server for you.

There’s a howto in the works for this example buildout, based on excellent presentations given by Nate Aune. At the Plone Symposium 2009 sprint in Sorrento I’ve helped a bit with the tutorial and got my first live experience with Deliverance. The tutorial is still a work in progress.  This blog post is mainly a collection of things I’ve run into so far, not a structured “do this, do that”.

One of deliverance’s requirements is python lxml, which in turn needs the libxml2 and libxslt libraries. Especially on Mac OS X these libraries can cause a lot of headaches, hence the z3c.recipe.staticlxml recipe to compile them static and not use the OS X system libraries.

One caveat here is that the recipe depends on zc.recipe.cmmi to compile the libraries but fails to create a ‘downloads’ directory in your main buildout. So create the downloads dir or make sure it’s in your buildouts svn repository.

Once you have the buildout running you can start the Plone site with bin/instance, and deliverance with “bin/deliverance-proxy rules.xml”

Default transforms

Deliverance does some default actions without you specifiying them in the rules.xml . This bit us while experimenting when we wanted to override which javascript/css was copied from the Plone site into the theme. You can suppress these default actions using the  suppress-standard=”1″ attribute on a rule container so that you can specify your own.

Switching themes

Our deliverance project at work needs a slightly different theme for the homepage and second level pages. In the rules.xml you set the default theme to use with:

<theme href="/static/index.html" />

Between the ruleset as the default template to use for merging your content into. Rules can be specified for certain classes only.

<theme href="/static/index.html" />
<match path="regex:^/$"/>
<rule class="homepage>
<theme href="/static/homepage.html"/>
</rule>


This will use /static/index.html as the default page, but for the exact path “/” switch to static/homepage.html . Note that you can use hints like ‘regexp:’ or ‘exact:’ in any rule attribute that does string matching. Without this ‘regexp’ the path attribute in the match action would have matched any request since the default for path attributes is prefix matching.

Feeding javascript/css into the theme

Putting content from the plone site in to a static theme is rather straighforward, excitement starts when you also want to provide Plone’s editing functionality in the deliverance served site.

Plone uses quite some css and javascript for the content editing. A standard Plone site will not include ‘authoring.css’ for anonymous users but add this into the head with an  @import when a user is logged in. Rules for this are set in the css_registry tool in the Plone site.

Ideally you would not want to use any css or javascript from the Plone site, but to support the editing styles you then have to select and copy everything necessary from the Plone styles in your static template. And anonymous visitors would still receive all the unnecessary authoring css.

Deliverance has some fancy CSS3 selector action that let you check on attributes in tags in the content and copy them for the theme. So could use

<prepend content="link[href *= 'authoring']" theme='children:/html/head' />

to copy a link tag with an href attribute to something with ‘authoring’ in the head section.  But this will only work if you set the rendering type of the css resource in the css_registry to ‘link’. By default it is set to import, but you cannot select this @import with the help of CSS3 selectors.

Plone also cleverly merges css resources registered in the css_registry into single files with a new name including a hash, for example ‘authoring-cachekey0936.css’.  This file then contains the authoring.css and every css below this file in the registry if it has marked as mergeable and the same rendering type. If you want to include ONLY authoring.css with Deliverance you will also have to turn off ‘merging’.

To make this easier we have still created a Plone theme for the site, even though Deliverance is sitting in front doing the main themeing. The plone theme doesn’t do much theming, but settings like these are easy to configure on the filesystem using Generic Setup in your theme’s profiles/default/cssregistry.xml

Where’s the code?

Once we’re done with the initial version of  project we are working on now, the buildout and theme will be released on the Plone collective to serve as another example. The more examples, the merrier.

Other resources:

As stated there are a lot of individual blog posts online, and the official documentation site obviously to begin with. Two  gems I found:

Macadames Blog : Tips on {SERVER_NAME} in your rules, apache virtualhost config and integration with PyQuery.

Ian Bickings’ PloneConf 2008 Deliverance presentation notes. These are buried in an svn sandbox but provide some very cool real world use cases with solutions and rule examples.

This one had biten me a few months ago and now again. A customer project includes the PloneSurvey product and downloads the package from www.plone.org plone.recipe.distros . No it doesn’t. If you try to download the package you get a 0 bytes tar file. Probably a left over from the file migration in the Plone Software Center.

http://plone.org/products/plone-survey/releases/1.2.0/plonesurvey.tar (don’t use this)

And buildout doesn’t complain. You only spot it later when you see Broken product references in the instance logs and see a “Product Removed” notice in the portal_quickinstaller in a site where it still was installed. Sigh.