Aug 2 2010

Memory monitoring with supervisor, memmon and and zc.buildout

UPDATE: I’vechecked in a fix on collective.recipe.supervisor trunk, hopefully it will be picked up with release 0.12.

I’ve been tinkering a bit with process memory monitoring for applications which are started and managed with supervisord. If you want Memmon or status checking you will need the ‚superlance’ package added.

Since we are using buildout to generate all our setups I’m using collective.recipe.supervisor to generate the supervisord.conf. Adding
plugins = superlance
eventlisteners =
Memmon TICK_60 ${buildout:bin-directory}/memmon [-p instance=400MB]

Does not get the monitor running though. When starting supervisord the memmon process is exitting to soon and stopping. Running memmon on the command line with the correct options gets it running though.

After some more trial and error and searching online the culprit seems to be a recent change with recentsupervisor releases. this is what the recipe generated for the eventlistener:

[eventlistener:Memmon]
command = /Users/fred/buildouts/officious/bin/memmon -p instance=400MB
events = TICK_60
process_name=Memmon
environment=SUPERVISOR_USERNAME=myusername,
¬ SUPERVISOR_PASSWORD=mypassword,
¬ SUPERVISOR_SERVER_URL=‚http://127.0.0.1:12345

The values in the environment line should be quotes though. So changing that line to

environment=SUPERVISOR_USERNAME='myusername’,
¬ SUPERVISOR_PASSWORD='mypassword',
¬ SUPERVISOR_SERVER_URL='http://127.0.0.1:12345'

makes memmon start up correctly. I’ll see if I can get this reported/fixed in the recipe this week.(updated on trunk).

btw. (¬ ) is linebreak, but should be continued on the previous line


Oct 31 2009

Building Plone 4.0 core-dev on OS X Leopard 10.5

update 1: there is python26 in MacPorts.

After the great Plone Conference 2009 that was held from wednesday to friday we are now sprinting two more days. I guess there’s more than 200 people sprinting on saturday and sunday, improving Plone in all area’s: documentation, code, add-ons, you name it.

I gave up on getting plone 4.0 coredev running on my mac a few months ago after five minutes when something started to complain, but with all the gread minds around me in Budapest and a slide show by Maurits I wanted to get it running now. But it’s still painfull for some reason and I have I have no clue why it works for everybody else but not for me. Anyway, here are the roadblocks I bumped into installing it on OS X Leopard 10.5 .

Python 2.6

You need python 2.6 to run Plone 4.  OS X leopard comes default with an older python 10.5. For doing my Plone 3.X projects I use the python2.4 from macports, which works great, but macports for Leopard doesn’t have a python 2.6 package. Snow Leopard raised the bar by including support for both 32bit and 64bit in python, and causing a lot of pain to build other python versions from source, because they might link to the wrong 32bit or 64bit support libraries.

Florian Schulze created a buildout builds python 2.4/2.5 and 2.6 for you, paying attention to all the dependencies. It’s major goal was to simplify install on Snow Leopard but it works on Leopard 10.5 as well. Or: it should, because it doesn’t for me, it stops when trying to compile python 2.4. Solution: remove all references to python 2.4 and 2.5 from the buildout.cfg : you only need python 2.6 for Plone 4 and the other versions are already there from macports or the system python.  Hurdle one taken, I have a python 2.6.

Distribute madness

After fetching the Plone 4 development branch from http://svn.plone.org/svn/plone/buildouts/plone-coredev/branches/4.0 ,  doing a python2.6 bootstrap.py and running bin/buildout, the buildout coun’t install mr.developer.

# ImportError: <module ’setuptools.dist’ from ‘/Users/fred/buildouts/pythons/python-2.6/lib/python2.6/site-packages/distribute-0.6.3-py2.6.egg/setuptools/dist.pyc’> has no ‘check_packages’ attribute
# An error occured when trying to install mr.developer 1.1.Look above this message for any errors thatwere output by easy_install.

Now what. For some reason the included Distribute version was still stuck at 0.6.3, when 0.6.6. is the most recent version. So I created a python  2.6 virtualenv from my pythons 2.6 buildout (virtualenv-2.6 is created in the bin folder there) and installed  0.6.6 by using the easy_install from the newly created virtualenv. By the way, distribute is the succesor to the setuptools package.

After having a virutalenv with up to date distribute I used it to bootstrap my Plone 4 coredev buildout again and run bin/buildout. jay!

mr.developer is drunk

Buildout finished with a report that mr.developer couldn’t find some sources. You can list them by running bin/develop status, they have exclamation marks before them.  In particular, Products.CMFQuickInstaller and Products.PlacelessTranslationService were not installed by mr.developer in the src/ subfolder. I checked paths, checked with other sprinters, did an svn update, reran bin/buildout, nothing.  In the end I just did a manual svn checkout of the two packages with the links provided in the sources.cfg from the Plone4 core-dev in the src folder.

Swallow the blue PILl

bin/instance fg to start Plone 4 and…… we have no PIL module. Ah, right, I created a virtualenv without site-packages for the python2.6 to install Distribute. Been there, done that. There’s an easy way to add the Python Imaging (alias PIL) to your virtualenv by running easy_install http://dist.plone.org/thirdparty/PILwoTk-1.1.6.4.tar.gz .

And we’re up and running with a Plone 4 test buildout on OS X Leopard 10.5 . Finally.

Update 1

Apparently I had a serious headache or my macports was messed up when I search for Python 2.6 in macports two weeks ago. Alec Mitchell points out it has been there for a long time, just search for with port search  python26. After installing python 2.6 with sudo port install python26 and not forgetting to install py26-pil as well, all of the above mentionned problems with distribute and mr.developer are gone. Pfew.

plone4


Jul 8 2009

Deliverance theming for Plone with buildout

We are actively using Deliverance at Zest Software to theme a default Plone site  with a completely new skin. With Deliverance you run a ‘transformation’ server in front of your website like a proxy, but as an added bonus (or more as its intended purpose) you select parts of pages that the Plone site serves and insert these snippets like the content, main navigation and support menu’s  into placeholders from a static html template with css and images on the file system.

Deliverance is being experimented with and put to production use by many organisations/people already . There are quite some blog posts and notes on the web, but Documentation and howto’s are still below the level of  more mature (read: older) projects. The official deliverance 0.3 documentation gives a thorough overview of commands and transformation, but doesn’t present much real world usage or a full ‘howto’. This is not surprising since deliverance is so general in nature (fetch, transform/merge and deliver). Your individual documentation needs really depend on the sources from which you want to merge.

Installing with buildout

We are using buildout to develop all our Plone Sites. In the Plone Collective there’s an example buildout ‘deliverancedemo’ that sets up a Plone site and deliverance server for you.

There’s a howto in the works for this example buildout, based on excellent presentations given by Nate Aune. At the Plone Symposium 2009 sprint in Sorrento I’ve helped a bit with the tutorial and got my first live experience with Deliverance. The tutorial is still a work in progress.  This blog post is mainly a collection of things I’ve run into so far, not a structured “do this, do that”.

One of deliverance’s requirements is python lxml, which in turn needs the libxml2 and libxslt libraries. Especially on Mac OS X these libraries can cause a lot of headaches, hence the z3c.recipe.staticlxml recipe to compile them static and not use the OS X system libraries.

One caveat here is that the recipe depends on zc.recipe.cmmi to compile the libraries but fails to create a ‘downloads’ directory in your main buildout. So create the downloads dir or make sure it’s in your buildouts svn repository.

Once you have the buildout running you can start the Plone site with bin/instance, and deliverance with “bin/deliverance-proxy rules.xml”

Default transforms

Deliverance does some default actions without you specifiying them in the rules.xml . This bit us while experimenting when we wanted to override which javascript/css was copied from the Plone site into the theme. You can suppress these default actions using the  suppress-standard=”1″ attribute on a rule container so that you can specify your own.

Switching themes

Our deliverance project at work needs a slightly different theme for the homepage and second level pages. In the rules.xml you set the default theme to use with:

<theme href="/static/index.html" />

Between the ruleset as the default template to use for merging your content into. Rules can be specified for certain classes only.

<theme href="/static/index.html" />
<match path="regex:^/$"/>
<rule class="homepage>
<theme href="/static/homepage.html"/>
</rule>

This will use /static/index.html as the default page, but for the exact path “/” switch to static/homepage.html . Note that you can use hints like ‘regexp:’ or ‘exact:’ in any rule attribute that does string matching. Without this ‘regexp’ the path attribute in the match action would have matched any request since the default for path attributes is prefix matching.

Feeding javascript/css into the theme

Putting content from the plone site in to a static theme is rather straighforward, excitement starts when you also want to provide Plone’s editing functionality in the deliverance served site.

Plone uses quite some css and javascript for the content editing. A standard Plone site will not include ‘authoring.css’ for anonymous users but add this into the head with an  @import when a user is logged in. Rules for this are set in the css_registry tool in the Plone site.

Ideally you would not want to use any css or javascript from the Plone site, but to support the editing styles you then have to select and copy everything necessary from the Plone styles in your static template. And anonymous visitors would still receive all the unnecessary authoring css.

Deliverance has some fancy CSS3 selector action that let you check on attributes in tags in the content and copy them for the theme. So could use

<prepend content="link[href *= 'authoring']" theme='children:/html/head' />

to copy a link tag with an href attribute to something with ‘authoring’ in the head section.  But this will only work if you set the rendering type of the css resource in the css_registry to ‘link’. By default it is set to import, but you cannot select this @import with the help of CSS3 selectors.

Plone also cleverly merges css resources registered in the css_registry into single files with a new name including a hash, for example ‘authoring-cachekey0936.css’.  This file then contains the authoring.css and every css below this file in the registry if it has marked as mergeable and the same rendering type. If you want to include ONLY authoring.css with Deliverance you will also have to turn off ‘merging’.

Plone css_registry

To make this easier we have still created a Plone theme for the site, even though Deliverance is sitting in front doing the main themeing. The plone theme doesn’t do much theming, but settings like these are easy to configure on the filesystem using Generic Setup in your theme’s profiles/default/cssregistry.xml

Where’s the code?

Once we’re done with the initial version of  project we are working on now, the buildout and theme will be released on the Plone collective to serve as another example. The more examples, the merrier.

Other resources:

As stated there are a lot of individual blog posts online, and the official documentation site obviously to begin with. Two  gems I found:

Macadames Blog : Tips on {SERVER_NAME} in your rules, apache virtualhost config and integration with PyQuery.

Ian Bickings’ PloneConf 2008 Deliverance presentation notes. These are buried in an svn sandbox but provide some very cool real world use cases with solutions and rule examples.


Jun 19 2009

zc.buildout and broken product distros

This one had biten me a few months ago and now again. A customer project includes the PloneSurvey product and downloads the package from www.plone.org plone.recipe.distros . No it doesn’t. If you try to download the package you get a 0 bytes tar file. Probably a left over from the file migration in the Plone Software Center.

http://plone.org/products/plone-survey/releases/1.2.0/plonesurvey.tar (don’t use this)

And buildout doesn’t complain. You only spot it later when you see Broken product references in the instance logs and see a “Product Removed” notice in the portal_quickinstaller in a site where it still was installed. Sigh.


May 11 2009

Where to start

That’s what I have been asking myself for the last year or so after I decided to rent a virtual slicehost and set up a blog. I still have  one and a half hour before I have to leave for the airport to the European Plone Symposius 2009, so let’s get this thing online. Me, procrastinating? Nah….

What a mess with the maximum luggage weight you can bring with you. Just my laptop in a sleeve is already 3 kg. So that’s why the new netbooks are so popular…  And would a Canon EOS 350 be a ‘normal format’ photo camera according to Transavia? Time to get my stuff packed (again).