Recently some of our Varnishes unexepectly quit. Checking the supervisor logs gives an ‘expected’ shutdown state, the same message you see when you shut down a process manually with supervisorctl. But we didn’t do that.

So: A) why is varnish quitting and B) Why doesn’t supervisor restart it?

To start with (B): by default supervisor thinks exit codes 0 and 2 are ‘expected’ codes on which a process isn’t restarted. You can change this by adding

exitcodes = 0

to your supervisord.conf or adding it as an extra setting like this in your buildout if you are using collective.recipe.supervisor:

30 varnish (exitcodes=0) ${buildout:directory}/bin/varnish

Now for (A). I haven’t nailed it down exactly yet, but most buildout examples (and our production servers are still using Varnish 2.0.X, where the cli_timeout by default is 5 seconds. This is a timeout for communication between parents and worker processes in the varnish setup. I’ve found this bug ticket #329 where the cli_timeout by default has been upped to 10 seconds, because some setups might timeout when the system is under heavy load (as we found out later from our munin graphs, the systems were quite busy around the time that Varnish exitted).

I haven’t been able to find much documentation yet on exit code 2 from varnish and I’m not sure yet if if is also sometimes used for ‘graceful’ stops. Which would mean that supervisor will not let the process end when it should, but hopefully this will solve our varnish exits for now.

If you want a newer Varnish in your buildout with the default cli_timeout set to 10 seconds, try plone.recipe.varnish 1.1b1 . One warning though: On my OS X 10.6 developer machine I had to install pcre using macports before varnish would build. All my tricks to get a pcre compiled in from source in my buildout cost me 100 grey hairs and half a day of trial and error without succes. But that’s something for another blog post.

Did you run into similar problems and/or have tips or other solutions? Please let me know.

I’ve been tinkering a bit with process memory monitoring for applications which are started and managed with supervisord. If you want Memmon or status checking you will need the ‚superlance’ package added.

Since we are using buildout to generate all our setups I’m using collective.recipe.supervisor to generate the supervisord.conf. Adding
plugins = superlance
eventlisteners =
Memmon TICK_60 ${buildout:bin-directory}/memmon [-p instance=400MB]

Does not get the monitor running though. When starting supervisord the memmon process is exitting to soon and stopping. Running memmon on the command line with the correct options gets it running though.

After some more trial and error and searching online the culprit seems to be a recent change with recentsupervisor releases. this is what the recipe generated for the eventlistener:

[eventlistener:Memmon]
command = /Users/fred/buildouts/officious/bin/memmon -p instance=400MB
events = TICK_60
process_name=Memmon
environment=SUPERVISOR_USERNAME=myusername,
¬ SUPERVISOR_PASSWORD=mypassword,
¬ SUPERVISOR_SERVER_URL=‚http://127.0.0.1:12345

The values in the environment line should be quotes though. So changing that line to

environment=SUPERVISOR_USERNAME='myusername’,
¬ SUPERVISOR_PASSWORD='mypassword',
¬ SUPERVISOR_SERVER_URL='http://127.0.0.1:12345'

makes memmon start up correctly. I’ll see if I can get this reported/fixed in the recipe this week.(updated on trunk).

btw. (¬ ) is linebreak, but should be continued on the previous line

You do it again with a reload. Nothing. You do it again. It moves up one spot. That’s not good enough… If you disable javascript in your browser you will see two arrows, this is the plain HTML fallback. it calls an url on the folder that ends with ‘mainfolder/folder_position?position=up&id=site-content’ to move the object up in the folder. But what’s going on with this non deterministic ordering?

My colleague Maurits van Rees had an interesting theory that seems plausible enough for me at the moment: in all the sites I have experienced thes ordering problems there’s LinguaPlone installed because the site is multilingual. And one of the tricks LinguaPlone has up it’s sleeves is that it will filter out all objects in folder listings and navigation that are in a different language than the language you’re viewing the site in.  So what is happening when you move an object one position up in a Dutch/English site where there are not 4, but actually 8 folders in the Plone site root? Nothing. Only after moving up 5 times you will notice any difference, because you have to pass 4 ‘hidden’ Dutch folders first if you are trying to re-order the English objects.

When using drag and drop the javascript involved is a bit more intelligent and tries to move the object for example 4 positions if you drag it 4 places above. But as you can see that doesn’t help in this case. Imagine how many times you have to move an item when there are 5 or 6 translations in a site, each with 5 folders in the main site root.

I’ve seen this happening in several Plone 3 sites with LinguaPlone 2.2/2.4 Hard to tell what/who is to blame or how to fix this, but it’s annoying at least.

After the great Plone Conference 2009 that was held from wednesday to friday we are now sprinting two more days. I guess there’s more than 200 people sprinting on saturday and sunday, improving Plone in all area’s: documentation, code, add-ons, you name it.

I gave up on getting plone 4.0 coredev running on my mac a few months ago after five minutes when something started to complain, but with all the gread minds around me in Budapest and a slide show by Maurits I wanted to get it running now. But it’s still painfull for some reason and I have I have no clue why it works for everybody else but not for me. Anyway, here are the roadblocks I bumped into installing it on OS X Leopard 10.5 .

Python 2.6

You need python 2.6 to run Plone 4.  OS X leopard comes default with an older python 10.5. For doing my Plone 3.X projects I use the python2.4 from macports, which works great, but macports for Leopard doesn’t have a python 2.6 package. Snow Leopard raised the bar by including support for both 32bit and 64bit in python, and causing a lot of pain to build other python versions from source, because they might link to the wrong 32bit or 64bit support libraries.

Florian Schulze created a buildout builds python 2.4/2.5 and 2.6 for you, paying attention to all the dependencies. It’s major goal was to simplify install on Snow Leopard but it works on Leopard 10.5 as well. Or: it should, because it doesn’t for me, it stops when trying to compile python 2.4. Solution: remove all references to python 2.4 and 2.5 from the buildout.cfg : you only need python 2.6 for Plone 4 and the other versions are already there from macports or the system python.  Hurdle one taken, I have a python 2.6.

Distribute madness

After fetching the Plone 4 development branch from http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.0 ,  doing a python2.6 bootstrap.py and running bin/buildout, the buildout coun’t install mr.developer.

Now what. For some reason the included Distribute version was still stuck at 0.6.3, when 0.6.6. is the most recent version. So I created a python  2.6 virtualenv from my pythons 2.6 buildout (virtualenv-2.6 is created in the bin folder there) and installed  0.6.6 by using the easy_install from the newly created virtualenv. By the way, distribute is the succesor to the setuptools package.

After having a virutalenv with up to date distribute I used it to bootstrap my Plone 4 coredev buildout again and run bin/buildout. jay!

mr.developer is drunk

Buildout finished with a report that mr.developer couldn’t find some sources. You can list them by running bin/develop status, they have exclamation marks before them.  In particular, Products.CMFQuickInstaller and Products.PlacelessTranslationService were not installed by mr.developer in the src/ subfolder. I checked paths, checked with other sprinters, did an svn update, reran bin/buildout, nothing.  In the end I just did a manual svn checkout of the two packages with the links provided in the sources.cfg from the Plone4 core-dev in the src folder.

Swallow the blue PILl

bin/instance fg to start Plone 4 and…… we have no PIL module. Ah, right, I created a virtualenv without site-packages for the python2.6 to install Distribute. Been there, done that. There’s an easy way to add the Python Imaging (alias PIL) to your virtualenv by running easy_install http://dist.plone.org/thirdparty/PILwoTk-1.1.6.4.tar.gz .

And we’re up and running with a Plone 4 test buildout on OS X Leopard 10.5 . Finally.

Update 1

Apparently I had a serious headache or my macports was messed up when I search for Python 2.6 in macports two weeks ago. Alec Mitchell points out it has been there for a long time, just search for with port search  python26. After installing python 2.6 with sudo port install python26 and not forgetting to install py26-pil as well, all of the above mentionned problems with distribute and mr.developer are gone. Pfew.

Deliverance is being experimented with and put to production use by many organisations/people already . There are quite some blog posts and notes on the web, but Documentation and howto’s are still below the level of  more mature (read: older) projects. The official deliverance 0.3 documentation gives a thorough overview of commands and transformation, but doesn’t present much real world usage or a full ‘howto’. This is not surprising since deliverance is so general in nature (fetch, transform/merge and deliver). Your individual documentation needs really depend on the sources from which you want to merge.

Installing with buildout

We are using buildout to develop all our Plone Sites. In the Plone Collective there’s an example buildout ‘deliverancedemo’ that sets up a Plone site and deliverance server for you.

There’s a howto in the works for this example buildout, based on excellent presentations given by Nate Aune. At the Plone Symposium 2009 sprint in Sorrento I’ve helped a bit with the tutorial and got my first live experience with Deliverance. The tutorial is still a work in progress.  This blog post is mainly a collection of things I’ve run into so far, not a structured “do this, do that”.

One of deliverance’s requirements is python lxml, which in turn needs the libxml2 and libxslt libraries. Especially on Mac OS X these libraries can cause a lot of headaches, hence the z3c.recipe.staticlxml recipe to compile them static and not use the OS X system libraries.

One caveat here is that the recipe depends on zc.recipe.cmmi to compile the libraries but fails to create a ‘downloads’ directory in your main buildout. So create the downloads dir or make sure it’s in your buildouts svn repository.

Once you have the buildout running you can start the Plone site with bin/instance, and deliverance with “bin/deliverance-proxy rules.xml”

Default transforms

Deliverance does some default actions without you specifiying them in the rules.xml . This bit us while experimenting when we wanted to override which javascript/css was copied from the Plone site into the theme. You can suppress these default actions using the  suppress-standard=”1″ attribute on a rule container so that you can specify your own.

Switching themes

Our deliverance project at work needs a slightly different theme for the homepage and second level pages. In the rules.xml you set the default theme to use with:

<theme href="/static/index.html" />

Between the ruleset as the default template to use for merging your content into. Rules can be specified for certain classes only.

<theme href="/static/index.html" />
<match path="regex:^/$"/>
<rule class="homepage>
<theme href="/static/homepage.html"/>
</rule>


This will use /static/index.html as the default page, but for the exact path “/” switch to static/homepage.html . Note that you can use hints like ‘regexp:’ or ‘exact:’ in any rule attribute that does string matching. Without this ‘regexp’ the path attribute in the match action would have matched any request since the default for path attributes is prefix matching.

Feeding javascript/css into the theme

Putting content from the plone site in to a static theme is rather straighforward, excitement starts when you also want to provide Plone’s editing functionality in the deliverance served site.

Plone uses quite some css and javascript for the content editing. A standard Plone site will not include ‘authoring.css’ for anonymous users but add this into the head with an  @import when a user is logged in. Rules for this are set in the css_registry tool in the Plone site.

Ideally you would not want to use any css or javascript from the Plone site, but to support the editing styles you then have to select and copy everything necessary from the Plone styles in your static template. And anonymous visitors would still receive all the unnecessary authoring css.

Deliverance has some fancy CSS3 selector action that let you check on attributes in tags in the content and copy them for the theme. So could use

<prepend content="link[href *= 'authoring']" theme='children:/html/head' />

to copy a link tag with an href attribute to something with ‘authoring’ in the head section.  But this will only work if you set the rendering type of the css resource in the css_registry to ‘link’. By default it is set to import, but you cannot select this @import with the help of CSS3 selectors.

Plone also cleverly merges css resources registered in the css_registry into single files with a new name including a hash, for example ‘authoring-cachekey0936.css’.  This file then contains the authoring.css and every css below this file in the registry if it has marked as mergeable and the same rendering type. If you want to include ONLY authoring.css with Deliverance you will also have to turn off ‘merging’.

To make this easier we have still created a Plone theme for the site, even though Deliverance is sitting in front doing the main themeing. The plone theme doesn’t do much theming, but settings like these are easy to configure on the filesystem using Generic Setup in your theme’s profiles/default/cssregistry.xml

Where’s the code?

Once we’re done with the initial version of  project we are working on now, the buildout and theme will be released on the Plone collective to serve as another example. The more examples, the merrier.

Other resources:

As stated there are a lot of individual blog posts online, and the official documentation site obviously to begin with. Two  gems I found:

Macadames Blog : Tips on {SERVER_NAME} in your rules, apache virtualhost config and integration with PyQuery.

Ian Bickings’ PloneConf 2008 Deliverance presentation notes. These are buried in an svn sandbox but provide some very cool real world use cases with solutions and rule examples.

http://plone.org/products/plone-survey/releases/1.2.0/plonesurvey.tar (don’t use this)

And buildout doesn’t complain. You only spot it later when you see Broken product references in the instance logs and see a “Product Removed” notice in the portal_quickinstaller in a site where it still was installed. Sigh.