Miles Crawford's Staff Page

Tuesday

Buliding Scalable Websites with Perl

Perl rules: Yahoo, Overture, Amazon, IMDB, Ticketmaster, and Citysearch are all on Perl. There's our street cred, because if nothing else, this shows that Perl scales. Apparently, they're not doing it just by throwing hardware at it either.

Caching the big one for increasing performance, and availble for use at many levels, database, interior to objects, up to page level. How much could we realistically cache? At the page level, it seems like very little to me... We are looking at a cache-on-demand method, but this still assumes that many people will be viewing the same dynamic page, which is just not true for most of our tools. Additional granularity can be achieved by caching only parts of a page, this might be closer to possible for us.

Memcached - lower level memory caching that works across servers.

Job Queuing The (lighter weight) web server handles easy stuff, and puts users in line for the hard stuff. What actually is the queue, though? An implementation to look at is Spread::Queue. It provides a very light and general API for wrapping referenced objects/functions to act as the worker. Weaknessess: no place in line or status messaging.

Search in Progress Fork a worker process and redir the client to a working page that refreshes every 2 seconds until the results have been deposited by the worker process. The worker can, of course, wait it's turn if needed.

Google for "stonehenge search in progress" for good example of this.

Keynote: Mono - Miguel de Icaza

We know Mono - it's an open source implementation of the .NET framework. It includes a virutal machine for running the software and tools for porting software from one platform to the other (.NET to Mono).

Ximian was geared towards bringing Linux to the Desktop, and while developing Evolution, they discovered they needed better tools.

Yay! The keynot speaker dissed Java!

.NET has a lot of strengths: C# is an improvement on Java and has a very clear, useful spec, but .NET can run and work with other languages as well.

Novell encouraged development by generating a large community by getting thier paid employees to do the boring and qa stuff while letting the community take on all the fun, highly visible stuff.

This has become pretty much an overview of the shiny beautiful tools being written for the Mono framework by Novell and the Ximian guys. Looks like some very pretty toys for the desktop, but very little to do with us.

PHP 5

This turned out to be more or less a cheerleading session, touting improvements:

Pretty sizable OO improvements

Class data/method hiding (public/private/protected)
Objects passed by reference instead of value by default
Builtin constructors
Strict function prototypes to enforce parameter types
Abstract classes
Auto class loading
Overloaded operators (this always struck me as possibly confusing but convienient)
Multiple inheritnce, but only via interfaces.

Exception handling, not something we take advantge of yet, even in the eval{} fashion.

XML handling has also gotten a facelift, with new XMLDOM APIs and features, as well as a SimpleXML library that allows XML docs to act as a builtin data structure. This looks pretty nice, done with a top->down object-like syntx.

Speaking of XML, built-in soap libraries make use of this feature (making PHP a decent choice for porlet writing, I would imagine).

Some slightly creepy tight-knit integration with SQLite... create references to bits of php code and reference them in SQL. RDMS-like behavior, but this is done in flat files by the interpreter so write locks stop reads too.

PHP to Perl bridge - pretty exciting, but I need to hear more... Oooooky this is just stupid. It shells a Perl interpreter.

There's some neat-looking data streaming stuff handled entirely by fopen (ssh sftp ftp socket file http (dav?) etc etc.)

Native use of Java objects. How efficient is this? Used to be a JVM per object, and the syntax looks the same...

Yay! php-gtk2!

The speaker is qualifying many of these features by saying "You probably shouldn't use this if you need to do lots of it, hammer it, you know..." So that sounds kinda familiar as far as PHP goes...

WSRP - Julie McNaught

The objective is to aggragate bits of functionality and content.

This is achieved by four main players:

Portlet - The portal is independant of the portlets in a big way - The only thing the portlet itself is responsible for is generating is generate markup fragments, and use a few specified CSS classes in so doing. The portlet itself is nothing more than a tiny markup generator. This is the first reason we can definitly write portlets.
Producer - Provides a collection of Portlets, who are probably running on the Producer machine.
Consumer - Gathers various Portlets into a useable collection of "boxes" (not neccesarily in a web portal).
End User - Ya. You get it.

All communiction beteen producer and consumer is via SOAP. This is the second reason why we can write portlets.

Now, if the user hits controls on a portlet, there is often supposed to be a state change on the producer. The consumer does not know the far-reaching effects of this, assuming they exist. The consumer therefore waits until the state change is signaled complete before it regathers the fragments from the producer. How sensitive is this? Does this happen only if there are multiple portlets gathered from the same producer? Will effects ever be felt across producers?

There are significant symantics that can signify the expiration, need to update, etc of markup blobs in response to user activities, but these cannot be lorded over from a central location (the consumer) so the portlets can remain independant. This results in no dependency checking between portlets.

Portlet registration or requests are separate from producer registration.

Both the producer makes specified changes to the portlet's markup, and the consumer knows how to deal with the producers content. Alternatively, the consumer can send hints to the producer, offloading the munging work and getting it ready for the consumer before transmission.

Similarly namespaces need munging treatment, they're not leaving it to chance. There are specific strings that the portlet can include that will be replaced by a unique string, or the consumer can, again, hint the producer to offload this task.

Sessions are given specific IDs by the portlet, and the consumer is required to pass these back with hits, so portlets can keep state. This can be done at a group level too. (I thought this was groups of people, but patrick saw it as a way to bridge sessions between portlets.)

Consumers may allow customizations. They do this by cloning the portlet. This creates a new instance of the portlet on the producer which up until now can serve up many users worth of content from a single instance.

Consumers can register with producers, for example to create authentication or payment relationships.

Fragment rules for portlet content exist, but are not at all restrictive. The group hopes that standards groups for various content types will define their own fragment standards.

A valid WebQ fragment is... ;)

Future improvements include looking into attachment handling to avoid processing all data with SOAP/WSRP protocols. Also, a discovery protocol to assist consumers in locating usable producers is in the works, or at least on the horizon.

My reactions: well, the portlet output is just HTML so there is no question at all: we can write portlets in Perl. The producer/consumer contact is via SOAP, so if we are allowed to be our own producer we will be able to this free of any hitches - Perl can do SOAP as well as anything. The real stumbling block would be whatever hoops we would have to jump through to communicate with a non-Perl producer, especially one not administered by us. The consumer is a non-issue, and could be COBOL written with Crayolas for all we care.

The other large issue is: can we write anything useful? This talk covered only the most basic concepts of URL rewriting and the way a change in a portlet can be indicated to other peer portlets, and our applications would require a lot more than what appears possible.

Right now the best I would expect to be able to produce is a really sweet mini version of the Catalyst Your Account screen: dynamic, personalizable and beautiful, with lots of links that took the user away from the portal to our full-fledged apps.