Resource Posts In Category
A lively discussion and exchange of information occurred recently on Hawk Wings, the blog site mostly devoted to news and resources for users of Apple’s terrific Mail program. A colleague at work sent me a message on Tuesday, excited when word on Hawk Wings started circulating about a “vacuum” process available for SQLite databases that appeared to dramatically speed up Apple Mail. He had tried the recommended vacuuming and definitely noticed peppier Mail performance. One thing led to another, and before I knew it, I’d become engrossed in developing and polishing up an AppleScript utility to automate a periodic vacuuming of my Mail, which I’m of course dubbing VacuumMail.
As the Hawk Wings discussion unfolded, we learned that Mail maintains an SQLite database called “Envelope Index” in your ~/Library/Mail folder, which gradually grows as the number of emails in your mailbox does. Natively, Mail performs no optimizations on this critical database, which contains pointers to all of your mail that become fragmented and somewhat disorganized over time. At the office, my Envelope Index file was over 100mb, and at home it’s about 30mb. SQLite offers a “vacuum” command that rewrites the Envelope Index, optimizing and reorganizing it for faster access. It sounds a bit like what happens when Mac OS X defragments your hard drive periodically.
At first, news of this function took the form of a shell command you can run in Terminal. It was quite interesting and exciting to see how the Mac users reading of this learned more about it as information was shared, and the command itself became more concise and precise as the day went on. Other users discovered that SQLite offers an “autovacuum” process that can do vacuuming without prompting, and I’m sure that’s a great thing as well. However, we also learned that vacuuming is a more robust and thorough optimizing of the file, since it actually analyzes and rewrites the whole thing, whereas autovacuuming acts only on a certain recent portion of mail pointers. The basic Terminal command turns out to be:
sqlite3 ~/Library/Mail/Envelope Index vacuum;
One very important lesson learned is that vacuuming is only available in the version of SQLite that comes with Mac OS X 10.4 (”Tiger”), and that “autovacuuming” is only available starting with 10.4.8. Of course, this should be fine with Panther users, since the problems I’ve had with Mail only really started with Tiger. It may be that use of Envelope Index itself started then as well.
So it wasn’t long before several people offered up AppleScripts to help automate the process, because who wants to fire up Terminal if you don’t have to? And by the end of the day a helpful reader had prepared and offered for download an Automator action you can use, with instructions for scheduling it through iCal.
Me, I saw an opportunity for combining some of the best ideas offered and for refining the way the AppleScript interacts with the Mail user. For one thing, I wanted a script that would run with no interaction at all, since sometimes that’s what I’d want. For example, if Mail isn’t running, the script doesn’t have to close Mail or start Mail up again afterwards. Further, I wanted a script I could schedule through Apple’s terrific, but still underused Launchd service, yet could also be run manually if necessary.
So, I took the opportunity to learn a little AppleScript and a little more about Launchd, as well as even a bit more about SQLite and Unix as well, to modify the script to my needs. Not being satisfied with that, I decided to also learn how to make an application bundle out of the script, so I could add a custom icon and maybe further refinements later on.
As my colleague noted, this whole episode is the latest that demonstrates what a great resource blog sites like Hawk Wings have become for Mac users. An outgrowth of one man’s personal interest and desire to develop a useful resource around one application (Apple Mail), Hawk Wings is an information resource that never could have existed prior to the advent of blogging. Mail is simply too narrow a topic to deserve a whole magazine or newsletter, and none of the traditional news outlets for Mac users would have been able or willing to devote the resources to the care and feeding of a website like Hawk Wings. So, my little foray is an attempt to give something back and to say “thanks” to Hawk Wings and its many readers.
The end result of my small effort is VacuumMail, which can be used as a regular application or automated through the included LaunchAgent for use with Launchd. I also took a few minutes to document the script and provide some information about Launchd and how to manage LaunchAgents and Services. The download package has three files: VacuumMail, the launch agent (which is just a .plist file), and a page of documentation.
Note: The following is the VacuumMail documentation included with the application package:
Â© 2007 by Leland Scott
Based on OptimizeMail by Sebastian Morsch, OptimiseEnvelope by pmbuko, and various other ideas and scraps of code on the HawkWings website.
VacuumMail is an AppleScript application that performs the “vacuum” command on Apple Mail’s underlying SQLite Envelope Index database. This must not be done while Mail is running, so this scriptlet first checks to see if Mail is running and if so, quits Mail. VacuumMail first notifies the user that it will be shutting Mail down, giving the user an opportunity to cancel the vacuum operation. This dialog window will close automatically, quit Mail, and commence vacuuming in 8 seconds if the user doesn’t click on the “Cancel” button.
VacuumMail is designed to be incorporated into an automated maintenance routine and run at regular intervals. According to the SQLite documentation, the vacuum command does a more thorough optimization of Mail’s database than the “autovacuum” setting that’s supported in Mac OS X beginning with version 10.4.8.
Alternatively, you can launch VacuumMail interactively if you need to do a manual run. I keep VacuumMail in my Finder toolbar, so it’s handy to launch.
If Mail was running when the process begins, VacuumMail will restart Mail when it’s finished. The vacuum process can take several minutes if your Envelope Index is very large, and in this first version VacuumMail doesn’t display any kind of progress indicator. However, when it’s finished, VacuumMail will display a dialog showing the size of your Envelope Index file before the latest vacuuming as well as afterwards. This dialog will close after a few seconds if you don’t close it manually.
VacuumMail can be installed anywhere on your system (unless you use the Launch Agent) but expects to find your ~/Library/Mail folder at that location. I did modify the shell commands so they’ll work with symbolic links, in case you, like me, store your Mail folder on a different partition but have it symbolically linked to your Library folder.
Using the VacuumMail LaunchAgent
The VacuumMail installer also includes a Launch Agent for launchd that you can use to automate the running of VacuumMail. This file, org.musingsfrommars.vacuumMail.plist, will be installed in your ~/Library/LaunchAgents folder. The installer will create this folder for you if you don’t already have one.
This agent assumes that you will put VacuumMail in your /Applications/Utilities/ folder, so that’s where this installer puts the application. If you want to put it somewhere else, you’ll need to edit the .plist file… either using the terrific, open source launchd editor Lingon, or in your favorite .plist editor. (I recommend one called PrefSetter, a free .plist editor that’s a big step up from the tool Apple distributes with its Developer Tools package.)
This agent will run VacuumMail every Tuesday afternoon at 1:00pm. I wanted to vacuum my mail weekly, and that time was chosen somewhat arbitrarily. If you want a different day and time, or a different time interval, it’s easy enough to change that behavior using Lingon. The other launchd option this agent utilizes is its ability to run in low priority mode. The launch agent runs with a “nice” priority setting of 15, so it will readily yield processor privileges to other applications on your system when it runs.
About Lingon and Launchd
Lingon is a graphical interface for creating launchd configuration files and controlling them through launchctl for Mac OS X Tiger. It was developed by Peter Borg and is available as open source from SourceForge. The Lingon website has good documentation for its use, as does the application itself.
The VacuumMail installer includes an option for you to install Lingon.
Launchd is a new system startup program Apple introduced in Tiger. The launchd daemon takes over many tasks from cron, xinetd, mach_init, and init, which are UNIX programs that traditionally have handled system initialization, called systems scripts, run startup items, and generally prepared the system for the user. And they still exist on Mac OS X Tiger, but launchd has superseded them in many instances. Apple’s use of launchd instead of the traditional UNIX programs provides a big performance boost to Tiger users. At any given time, only those daemons that are actually used are launched; combined with the fact that daemons can shut themselves down and be relaunched as needed means that launchd can reduce the average memory footprint of the system. For more information, see Apple’s excellent documentation on Launchd, from which this description is summarized. In addition, I noticed tonight that Lingon is one of the projects managed through the relatively new MacOS forge website, which is a SourceForge-like gathering place for Apple-initiated open source projects: Definitely checkout the Lingon site on MacOS forge for the latest and greatest developments with Launchd, or if you’d like to contribute some of your brains to the project.
VacuumMail is Â© 2007 by Leland Scott. It is released as freeware under creative commons license 2.5, which means you can modify it and use it freely except
- In commercial projects
- Without giving proper credit (Leland Scott)
Lingon is Â© 2006 by Peter Borg. It is released through SourceForge under the BSD license and can be used subject to those terms. It is included in the VacuumMail package with the generous permission of its author.
If you have feedback about VacuumMail, contact Leland Scott at llscotts [at] fastmail.fm. However, I won’t be able to provide support for the fundamental components of the application: SQLite, Apple Mail, Unix commands, AppleScript, Launchd, or Lingon. I suggest you read through the very interesting dialog about all of this on the HawkWings website, linked previously.
Peter Borg can be contacted through the Lingon website.
|0.9.6||4/18/07||This update fixes a bug in the 0.9.5 installer that inadvertently changes permissions to your /Users/Shared folder. This can cause conflicts with some applications such as iTunes that expect the /Users/Shared folder to be universally writeable. The 0.9.6 update ensures that /Users/Shared is set to the proper permissions after you run the VacuumMail installer. If you’ve already installed VacuumMail 0.9.5, you can probably fix the permissions problem by using DiskUtility to check and fix permissions on your system.|
|0.9.5||3/26/07||Provided an installer package for VacuumMail, and also included Lingon as an optional install. The installer puts Lingon and VacuumMail in your /Applications/Utilities folder, and the launch agent file in your ~/Library/LaunchAgents folder. (It creates the folder if you don’t already have one.)|
With growing interest and amazement, I read the back-and-forth argument between two long-time, highly respected Mac nerds yesterday on the subject of Mark Pilgrim’s decision to abandon Mac OS X for Ubuntu Linux. John Gruber is simply one of the best Mac writers there is, and regardless of what he has to say on a particular subject, you have to admire the elegance, precision, and logic of his writing. So when Gruber raised questions about the wisdom of Pilgrim’s move in a recent blog post, his large readership weighed in, and Pilgrim responded, you can be sure that a great many Mac users like me paid attention.
As usual, I agreed with nearly everything Gruber had to say, and the couple of niggles I have are not worth mentioning here since they would distract from the purpose of this article. And what is that purpose, you are wondering? Before I get to that, let me briefly summarize (if I dare) the exchange so far between Gruber and Pilgrim.
- Pilgrim has become fed up with Apple’s “closed”-edness. After 22 years as a sophisticated, high-end user, he’s decided Apple’s “closed” ecosystem of software and hardware is too closed for him. His primary concern is that the integrity of the data he stores in that ecosystem is at risk, because Apple doesn’t always document its data formats and doesn’t respect for long the proprietary formats it develops for storage. Pilgrim feels jerked around from one closed format to another and is tired of the data conversions and consequent data loss they inevitably entail.
- Gruber is surprised and a bit incredulous that Pilgrim would have suddenly been bitten by this bug. He agrees that closed formats aren’t good for long-term archival purposes, but questions whether losing his iTunes metadata and other format problems is worth chucking his expertise with the Mac operating system for something completely different. He points out that a good backup strategy is part of the solution to preserving precious content. He also devotes a large part of his response to criticizing the Mac blog writers who had knee-jerk reactions against Pilgrim’s decision, and who cited old “Mac is better than Windows because…” arguments without realizing the advances Windows has made since Windows XP (or 95, or whatever). Gruber argues against black-and-white thinking in general and for the very reasonable position of respecting other people’s choices even if you don’t agree with them.
- Pilgrim replies that Gruber missed his point and reemphasizes that his feeling “closed in” by proprietary formats has been coming on for a long time. Apple’s decision to abandon the widely used and understood mbox format for Mail was just the last straw. He feels betrayed that Apple switched formats in Tiger without informing its users, without providing them a way to back out, and without documenting the new format.
So why do I want to wander into this disagreement between two Macintosh heavyweights I don’t know, but greatly admire and respect? As I read their separate articles, I saw something with my Martian eyes that may not be clear to them. What I saw wasn’t an OS switch story, but rather a love story.
I’m coming to believe that the human brain just isn’t very reliable. Long ago I concluded that humans would never understand their own behavior, simply because they’re not capable of analyzing behavior without affecting it. The mind is too complex, there are too many variables that define behavior, and how can you stand outside human-ness and study it without reflecting your own beliefs and preconceptions? It just can’t be done, which is why we’ve made so little progress in psychotherapy and instead are becoming more and more dependent on the “objective” injection of drugs (which we also don’t fully understand).
So the prospect of someone as intelligent and knowledgeable (those are different things) as Mark Pilgrim abandoning an OS as highly evolved as Mac OS X over a file format issue is incredible. This is clearly an emotional response, and that’s the only way I can understand it. As a heavily invested Apple user myself, I have become incensed at the way Apple often treats its customer base. I’ve only been a Mac user for 10 years–less than half of Pilgrim’s 22–but I definitely feel Apple “owes me” something for my “loyalty”, especially after not abandoning the platform in the sad years of the late 1990’s when Apple lost its way. Like Pilgrim–and Gruber–I have many criticisms of the Apple platform and software, and if that Automator action interrupts my work one more time I’m gonna scream! (Why can’t it work in the background when it doesn’t require any input from me?) I have tried–and abandoned–and tried–and abandoned–using a local iDisk with .Mac so many times it’s not at all funny. Each time Apple says they’re improving webdav for .Mac, my hopes go up, and I try again. I’m currently trying again, in fact… we’ll see how long it lasts.
One of the advantages of being a Martian, though, is that I do have the ability to stand outside myself and see when I’m being silly. (It’s the antennae.) So I would recognize when my “fed-up”-ness was wresting control of my good judgment and rein it in. In this case, I think Pilgrim has failed to recognize that his beef isn’t with Mac OS X or the physical entity he calls his Mac, but rather it’s with the faceless corporate entity called Apple, run by its arrogant and righteous managers and programmers.
As Gruber points out, the question isn’t whether Apple is open or not, but whether they’re open enough. One of the things I value about Apple and its products is the new ideas they bring to computing and their willingness to take risks with new approaches. I’m a “love new stuff” kind of guy, so I welcome anything new with open arms. This can be a problem when the new thing turns out to be a skunk in disguise, but with Apple that’s pretty rare. As a New-loving guy, though, I realize that nothing lasts forever. New things always displace old things, and they’re only new for a short time.
As I reminded a web developer colleague of mine recently, when you’re in the business of building websites, you simply have to accept the fact that whatever you think you know today is not going to be sufficient 2 years from now. API’s change, languages change, programming techniques change, graphics technology changes, browser technology changes, hardware capabilities change… you name it! It’s all malleable, and you have to be able to roll with it.
This constant “newness” in computing means that if you want to play and create in the digital world, you have to be prepared to convert from one format to another many times over in the course of your lifetime, or risk leaving valuable creations behind, locked in some old file format (or hardware format) nothing can read anymore. (The alternative, of course, is to stick with easels and canvas and pencil and paper.) I can easily identify with Pilgrim’s concern here… all creative individuals can. What we make we want to keep (the good stuff, anyway), and we want to be able to enjoy it again 5 years from now, or whenever the mood strikes. (I better do something about that large reel of magnetic tape with the original copies of my 1978 recordings on it before it’s too late!) Like Pilgrim, this is a constant worry. I try not to go into any new technology or tool without understanding how I’ll get my content out again. If I find I can’t, I abandon the technology before I’ve invested more of myself than can be easily migrated manually.
A couple of recent examples from my world… Bloglines is a great RSS service, and after having used NetNewsWire for about a year, I switched because Bloglines had this very cool “clip” function. In Bloglines, you can easily “clip” an article and assign it to a folder. Sounds like a great way to archive content you’re interested in, no? No. It turns out Bloglines provides no way either to present that content except through their administrative interface, and no way to export the metadata the clips represent. So, bye bye Bloglines.
Del.icio.us is another terrific service for Web 2.0, and I still use it. But when a series of service interruptions meant I had no access to my bookmarks a few days last fall, I panicked. I used a free plugin to Wordpress called Mysqlicious to import all of my Del.icio.us content and metadata into a MySQL database, and I now keep a mirror of all my Del.icio.us bookmarks there.
In fact, these two conversions are what led to the evolution of Musings from Mars. The “library” of News, Resources, and Software I keep here is nothing more than my bookmarks, all grown up with a great deal more content and metadata than I could store in Del.icio.us or Bloglines. I’m completely at ease now, because relational databases and SQL are standard, well understood data stores and methods of retrieval. The content itself is simply Unicode text, in some cases tagged with HTML. As a web guy, I’m very comfortable with HTML as an archival storage method, and with standard graphics formats like GIF, JPEG, PNG, and TIFF to store the images.
So, what about some of the conversions Pilgrim has had trouble with? Mail, for example? I’ve had lots of fun with mail formats over the years, but it’s possible that the solutions that work for me just wouldn’t do it for him. First, regarding Apple’s new “closed” format that replaces mbox. Is this such a tragedy? I’m of the mind that the engineers at Apple are a lot smarter than me about this kind of thing, and if they felt the need to change mbox in order to optimize Spotlight, I say, go for it! Being able to search across all of my past email is the most important thing anyway, isn’t it? What’s the good of having your mail in an “open” format, if it’s not easy to search? Pilgrim’s beef seems very strange to me, especially after I did a couple of quick experiments this morning with the .emlx format. (Actually, it looks to me like Apple still uses mbox for the mailboxes themselves, and emlx for the individual mail items. In any case, .emlx refers to the mail items, not to the mailboxes.)
First, I tried a new (to me) piece of shareware called File Juicer just to see what would happen when I fed it a folder-full of .emlx files. I was pleasantly surprised to find that everything converted very neatly to .txt, .html, .gif, .jpg, etc files. File Juicer put each file type in its own folder at the end. Lo and behold, all those ads I’d trashed still had their HTML files gloriously preserved! All of the attachments were neatly dumped out for me. Now, this isn’t an email archive, but if it’s the content you want to get at, it’s certainly an easy way to do that.
Second, I used a tried-and-true piece of shareware called Emailchemy, which I had previously used when archiving my Exchange mail to Apple Mail last year. Emailchemy makes it wonderfully easy to convert from just about any email format to another. You just point it at your Mail folder, and Emailchemy will preserve the directory structure, setting up mbox files, Eudora files, Thunderbird files, and more, which you can then import into another mail program. If you want to use a Mail client interface for accessing your archived mail, this is a very easy way to do that. It didn’t take long to import my Apple Mail mail into the terrific Opera mail client this way.
Third, I went ahead and tried a piece of freeware I’d downloaded earlier this year called MHonArc, which is specifically designed to convert from email formats to HTML as an archival utility. Now, this is thinking outside the box, folks! MHonArc is a command-line utility, but there’s a Mac OS X interface (also free) that makes it easy to use without having to learn the command syntax. You just point the software at your mail files, and it converts them to linked HTML. After I combined my “sent” and “inbox” folder in the same archive, I really saw the wisdom of this approach. Since the software preserves threads, now I could easily find my replies and my receipts in the same thread! And honestly, I think data in HTML is pretty darn safe!
A last option I didn’t bother to try, but which I find also very compelling is a tool like MailSteward, which converts your email to a relational database. Now honestly, Mark, doesn’t this sound better than switching to Linux? For $50, you can sock your email in a safe database from which you can output text files, SQL files, mbox files (yes!), and print (PDF) files, and which you can search much more flexibly than any Mail client can.
See, Mark… it’s not Mail, and it’s not iTunes, and it’s not AppleWriter, or whatever. Slowly, over many years of frustrations, you’ve developed a negative attitude toward Apple, and the bough has now broken. Apple is a lucky company in that they instill intense loyalty, verging on worship, in their users. But like love, loyalty has a dark side. Like someone we love who has betrayed us, Apple has a way of pissing off its loyal customers through neglect and indifference. Everyone who has used the Apple support forums has found them useful, but also quite cold. I have never ever seen an Apple employee step in to one of them to help people out. In fact, they seem to deliberately avoid doing so. Not good PR, in my view.
Apple and PR
A lot of Windows tech writers think Apple is great at PR, and that we all love Apple products because we love the packaging, or the advertising, or whatever. But Apple is actually pretty lousy at PR, except the very quiet kind at a very safe distance. Their website is wonderful… easy to use, inviting, attractive, informative, filled with tutorials and videos, and other fun stuff. The support site is great, too, but you never sense there’s anyone at Apple actually at home when you visit.
As an introvert personality, I can understand this: I like to provide interesting things and hope people enjoy them or find them stimulating. But I don’t want to shake hands with anybody or stand in a room filled with readers and give a talk. Still, you gotta admit it’s not a great PR approach.
When Apple does get noticed–its latest TV commercials, for example–they’re flashy and interesting and well made, but they aren’t going to change anyone’s minds. I don’t think the iPod ads ever sold an iPod, actually. What sold the iPod was friends meeting friends who showed them their iPod. It’s such a great product it “advertises itself.” If iPods were priced like PC’s, this would not have happened, but once Apple got the iTunes music store going on Windows, and an iPod for under $300, that’s when the gates really opened up for the iPod economy.
After having spent $7,000 in one year at my local Apple store, I became livid at a manager there who refused to give me my Federal employee discount, simply because I’d forgot to bring it to the store with me. I said, “Just look me up in your database.” He said, “We don’t have access to that information.” “What!? How can you provide top-notch customer service when you have no way of getting to know your customers?” He just shrugged his shoulders and acted like he didn’t know, didn’t care.
If Apple’s customers sometimes feel like they have a personal relationship with the company, then it can be a particularly bruising relationship for guys like Pilgrim who are digitally gifted and technologically savvy. It would be like being married to a spouse who you admire because they’re a little bit smarter than you. The spouse does many wonderful things and makes many fine decisions. But lots of decisions get made without consulting you. You argue about this, and she promises to get your opinion the next time before going off in a different direction. But then she doesn’t. Never does, in fact. Even when you think she’s brilliant, you bristle at her superior attitude. Finally, it’s too much for your ego, and off you go.
For “the rest of us,” Apple is simply brilliant so much of the time that we just forgive the few blunders. I personally stand in awe of so much about Mac OS X that I could give a rat’s ass about the mailbox format it uses. My issue is, “Make Mail play nice with Exchange 2000!” I’m delighted that Apple uses XML as a core data format, including the format for my iTunes metadata. I’m in love with QuickTime because it’s completely interoperable with open video and audio standards, and it’s extremely easy to convert from one format to another. Unlike Microsoft, Apple hasn’t tried to develop its own formats for the sake of controlling the standard and locking up the market. At least, I don’t believe that’s their motivation. This is why there is no Apple video codec, or audio “codec,” or graphics “codec.” Apple uses PostScript (PDF) as the basis for its graphics engine rather than developing one of its own. Isn’t that pretty darn open?
No, the beef isn’t with Apple’s openness or about conversion issues, which are generally very easy to work out. Actually, one of the risk factors for data that Pilgrim leaves out is the degree of support the format has from developers. To that extent, as Gruber says, the era of being at risk by using a Mac is now over, and if anything the pendulum is swinging the other way. There are so many developers building quality Mac OS X applications nowadays, that Cocoa is well on its way to being a factor that converts users to the Mac all by itself.
On the other hand, a format like mbox is used less often nowadays. Thunderbird doesn’t use it, and neither does Outlook, or Opera mail. Eudora has a shrinking slice of the pie, and Apple Mail’s slice is rising. I’m not saying mbox is at risk, but I wouldn’t count on it being around forever. It’ll be around only so long as there are developers willing to support it, which requires customers who are demanding it. Moving from one minority platform to an OS with even smaller support–especially when the platform you’re leaving supports everything the minority platform does–seems a little odd. Especially when the platform you’re leaving is more open than it has ever been before, as Gruber points out.
So odd, in fact, that I think it can’t be explained logically. None of the reasons Pilgrim gives make any sense. I’m not arguing for Mac OS X or against Linux here, I’m just saying a switch like this takes a great deal of effort, and why turn your world upside down over a change in mail formats? Especially when all you really have to do is switch mail clients. That, I could understand. I’ve done it often enough myself.
No, what we are witnessing is the end of a love affair. And when someone falls “out of love” with a company like Apple, there are no counselors the couple can turn to for help. So they do the only thing possible… a clean break, get away from everything that reminds you of the great things you accomplished with that beautiful spouse over the last 20 years.
You write up your reasons for breaking up, and everyone but you realizes you can’t see the full picture. But certainly I, like many Mac users in a similar position vis a vis Apple, mourn the breakup and wish it didn’t have to be so. Powerful emotions can close minds as surely as any proprietary format. And unlike that closed format, prying open a closed mind is nearly impossible.
I spent a few weeks in December 2005 investigating the universe of wiki software, and confirmed what I already suspected: It’s a very big universe with many wikis! It would be impossible to explore them all, so I first tried to come up with a short list of wiki engines to focus on. Fortunately, there are a number of excellent sites that attempt to provide matrices of wiki software functions and abilities. Here are a few I used and recommend:
- Good reviews of wiki software at onLamp.com, a site devoted to open-source LAMP products.
- This is a very thorough â€œchoice treeâ€ for wikis.
- Splitbrain, which makes Dokuwiki, has a good comparison page on wikis.
- Best of all, donâ€™t miss the new Wiki Matrix website, which evolved from a static HTML table matrix last fall.
After studying these various resources, I was able to narrow the list of wikis down to the following:
MediaWiki was the default choice, since I assumed it was probably the best of the lot, given its starring role in powering Wikipedia and just about every other high-profile wiki you encounter on the web. After a painless default installation of MediaWiki, I had the usual MediaWiki shell and did a few quick walk-throughs of the structure just to make sure all the plumbing was in place. It seemed to be, so I proceeded to install a few of the others from my short list.
In fairly quick succession, I installed Dokuwiki, PMwiki, and Tikiwiki, reviewed their documentation and capabilities, and did some basic configurations. They all seemed to be reasonably good, but none was noticeably superior, at first glance, to my initial configuration of MediaWiki. It seemed to make sense to stick with MediaWiki, given its large market share and equally large mind-share.
So, over a period of about 2 days, I began trying to configure MediaWiki to do some things beyond its default behavior–things I knew would be needed to provide a useful wiki for my target, non-technical clientele.
What a mess! I had spent 2 solid days without accomplishing much of anything toward setting up the desired wiki, which by the way was intended for use by a Federal organization that was interested in testing the use of wikis for developing and maintaining standard operating procedures for its divisions and branches.
Here is a summary of the problems I encountered with MediaWiki:
- Basic help on structured wiki markup was not available from within the software. In fact, no help files were loaded by default. Users are expected to create their own help pages.
- Basic help on structured wiki markup was not available from within the software. In fact, no help files were loaded by default. Users are expected to create their own help pages.
- The software’s documentation is terrible. The main problem is that there are so many sources of information, you get conflicting instructions. Many of the conflicts have to do with the various versions of mediawiki (1.3, 1.4, 1.5, etc).
- Creating simple navigation is quite difficult. One approach to navigation is to use â€œsub-pages,â€ but then forming links is tricky, and the page names include their parents by default. In other words, the relationships are discovered strictly by naming. Using piping, itâ€™s possible to make the link text look OK, but the titles on the pages are another issue.
- MediaWiki includes no basic, web-based administration tools at all. In fact, thereâ€™s no detection of sysadmin capability at all in the interface. To change the links in the Navigation box, for example, it turns out (after hours of hunting) that you are supposed to change the text in a page called Special:Allmessages. Not exactly intuitive, and it’s set up by default so as to be editable by anyone.
- Another useful navigation feature–breadcrumbs–donâ€™t exist, and they canâ€™t be created without custom coding. (Thereâ€™s an extension for this, but it only works in an older version of MediaWiki.)
- Skinning is also very difficult compared with the other wiki software I had looked at.
- A basic requirement for this project that I understood was not natively wiki-like was the need for some basic authentication and the ability to write-protect certain parts of the wiki tree for different groups. MediaWiki has a plugin for authentication, but it turns out that anyone who has administrator privileges can edit any part of the tree, and that wasn’t going to be sufficient in my security-conscious Federal agency.
After this experience, I decided to return to the drawing board, and take a second look at the short list packages. I also added a new one: Twiki. Itâ€™s written in Perl and uses flat files, but appears to be much more â€œmatureâ€ than some of the others.
In general, my impression after working with these various software packages is that wiki software is not nearly as â€œmatureâ€ as blog software. I was looking for an open-source wiki that would be as powerful as WordPress is in the blog world, while also being as easy to design, configure and administer as WordPress.
Twiki wasnâ€™t much better, and neither was MoinMoin, which I also ended up checking out (even though MoinMoin is written in Python, and I had no Python programmers to call on). Despite much positive press, MoinMoin has the same deficiencies as other wiki software. And what are those?
Basically, wikis were developed for use by programmers as a way of sharing information on software projects. They developed around a culture of highly sophisticated hacker-types who didnâ€™t need a lot of hand-holding when it came to navigation. The main concern was to allow rapid development of pages on a new topic, with automatic links to pages that hadnâ€™t yet been written (but which needed to be written). Wikis were designed to grow organically, as one writer filled in the blanks in anotherâ€™s page by adding information to it through hyperlinks, or as multiple writers contributed to fleshing out the details on a particular topic. In both cases, the result was to produce a decentralized information resource that relied primarily on search for finding things.
On Wikipedia today, itâ€™s become clear to those â€œin chargeâ€ that strong editorial oversight is needed to keep a wiki useful. For one thing, wikis donâ€™t automatically understand synonymous terms. One person may write a page that has a link to a new page called â€œWikiSystemsâ€, and another may already have filled in a page called â€œWikiSoftware.â€ Unless someone were watching â€œfrom above,â€ you could end up with two pages that covered pretty much the same ground.
Also, notice the terms â€œWikiSystemsâ€ and â€œWikiSoftware.â€ In wikis, the default way of linking is to write new pages in what is known as â€œcamel case:â€ Two words â€œmungedâ€ together, each having an initial cap. Wiki software is designed to recognize camel-cased terms and to automatically hyperlink them. Again, this is useful in its original conception, but itâ€™s not particularly intuitive for a nontechnical user base such as you would find in most business or government organizations.
Another shortcoming that many wikis donâ€™t handle well is authentication. Most wikis are designed to allow content editing by anyone. Most also allow administrators to restrict editing to registered users only. However, the ability to restrict access to certain pages to only certain people is not a native ability in most wiki systems.
Before I get around to describing the software I ultimately selected, I want to include my impressions of a few commercial software packages that have developed in the last year in an attempt to feed the growing market for wikis in corporate Intranets. One of the most well-known is Jotspot, an outsourced wiki system that can be purchased for a monthly fee. Jotspot is probably the most advanced wiki of this type, although since December there have been a fairly large number of newer entrants to the field, and itâ€™s possible that Jotspot has some good competitors by now. Jotspot is actually more of a full-blown Intranet than a wiki. Indeed, it shares this characteristic with Twiki, which branches out way beyond the central wiki functionality. Besides being a wiki, Jotspot (and Twiki) comes with a large number of plug-in applications that can be used for various Intranet functions (e.g., Project Management, Bug Reporting, Company Directory, Knowledge Base, Call Log Management, Blogging, Group Calendaring, Meeting Management, Polls and surveys, Personal to-do lists, etc.) The hosted version has a reasonable price tag, maxing out at $199 a month for unlimited users.
Jotspot also has an enterprise version for companies that want to host the software themselves. I set up a test wiki at Jotspot, and although it definitely has a lot to offer, it also isn’t nearly as configurable as one of the open-source packages. In addition, I felt certain I could find a perfectly good wiki package for my target organization without investing a lot of money.
Another impressive, hosted wiki-like system is Backpack, and I also set up a test there. However, Backpack is designed to work best as a personal wiki, rather than for collaboration. The same company also makes a web application called Basecamp that looks like an ideal solution for project management uses, but is not designed for documentation or knowledge management–the two main uses that this pilot wiki would be put to.
And if anyone was interested in a personal wiki, I don’t think you could do much better than Tiddlywiki, an amazing, rich-web interface â€œwiki on a stickâ€ that literally packs all of its information into a single portable file. It works an amazing amount of magic that could possibly be useful collaboratively, but that is designed to work best for individuals.
Finally, I looked at Projectforum, a commercial package that the customer was interested in. It turns out that Projectforum is not a wiki system, actually. Rather, it’s a discussion forum package (there are hundreds–possibly thousands–of such packages) that is trying to leverage the buzz around the term â€œwikiâ€ and RSS.
The critical difference is that a wiki is primarily a content management system, not a system for user discussions. MediaWiki uses the term â€œcollaborative editing,â€ because wikis typically have built-in discussion forums for each piece of content that gets added to the wiki. For example, if I post a Standard Operating Procedure on designing a website, readers would have the ability to create a discussion about that SOP. Also very important is the ability for users to interlink content into a growing content tree, producing in the end a very useful knowledge-base of information on a given topic.
Projectforum doesnâ€™t have those features, and is missing other standard wiki features as well. As its name implies, Projectforum is actually designed for project management rather than content/document management, and it excels at the collaborative discussion part of project management. In that sense, it is similar to Basecamp.
So after this market review, I had almost concluded that no wiki was really yet up to the challenge I was hoping to put it to, when I decided to try a relatively new, little known package called Wiclear. After reading through the website documentation, I tried to quell my growing excitement, because on paper at least, Wiclear was designed to overcome all of the shortcomings that were so obvious in all the wikis I’d tried.
Developed by a French programmer and modeled after a French blog system called Dotclear, Wiclear shares with nearly all other wikis the virtue of being open-source. Meaning, I can freely download the source code and install it. Wiclear is written using PHP, an increasingly popular web programming language, and the open source database MySQL. Since I happen to have some expertise in both, I felt comfortable with the prospect of possibly having to tweak the system to my requirements.
Indeed, after only 3 hours of work, I was able to configure Wiclear with all the basic requirements:
- Apply a customized style sheet
- Customize the section navigation
- Customize the page elements
- Customize the heading
- Set up test users
- Enter test content
- Set up appropriate help documentation for a wiki-nubi.
Compared with my experience with the other wiki software–in particular, with MediaWiki, Wiclear was very easy to work with. Furthermore, Wiclear had the following required features, some–but not all–of which were available in one or more of the other wiki systems.
- Browser-driven installation
- Web administration interface
- Easy templating
- Hierarchical page structure enforcing parent-child relationships between pages
- Individual page access controls through use of industry-standard ACLâ€™s (access control lists); the system provides an easy web-based interface for setting per-page permissions
- An automatically generated â€œsite planâ€–site map–for navigation
- Automatically generated â€œbreadcrumbsâ€
- Automatically generated â€œsub-page navigationâ€ (showing all child pages to the current one)
- Registered users can add comments about any page, whether they are the author or not. (This feature is configurable and is in fact a standard feature of most wiki systems.)
- Users can attach external files to individual pages (a relatively rare wiki feature, but one that I was sure would be â€œoohed and aahedâ€ at by my customer base.
- Enables user self-registration, and provides flexible User/group management tools.
- Provides a â€œPost New Contentâ€ feature thatâ€™s unique in wikiâ€™s, but extremely useful for adding new content to the tree.
- Usual features that made wikis so popular for collaborative editing in the first place:
- Page history
- Comparisons with and rollback to earlier pages
- Subscriptions by email
- RSS feeds
- List of recently changed pages
- â€œWhat links hereâ€ feature
- Simple editing system for easy content entry (with optional HTML entry), as well as an optional preview capability
Further, if my customers were ever to require the ability to support multiple languages, they could turn on one of Wiclearâ€™s most impressive features: built-in multilingual support.
The second tweak that might be of interest to readers was a default setting to automatically subscribe an author to the page he/she has written. This ensures that anyone who authors a page gets notified whenever it has been changed. (You cannot opt out of this feature, but you can always unsubscribe.) I hope this will take care of the worry over unauthorized edits, since it will be hard to not know when â€œyourâ€ page has changed, and quite easy to go in and fix any errors.
The author of Wiclear has steadily continued to improve the product. There have been 3 new releases since I installed Wiclear in late November 2005. In fact, the author has incorporated at least one of the features I requested after my initial configuration–namely, the ability to define a â€œrootâ€ page that could be ACL-protected against accidental damage. This was kind of important to give my customers the necessary comfort level to know that their part of the tree wouldn’t be uprooted someday, either advertently or inadvertently. I actually hand-coded the hack into Wiclear at the time, but the software’s author had finished integrating that function by January.
So far, I’m very pleased with my choice, and still relieved that I didn’t have to back out of the idea of testing the wiki waters for collaborative editing. Next comes the more difficult part–convincing users that this is a tool that can work for them rather than simply another complication to their working lives. Fortunately, there are several forward-thinking groups in the agency that are anxious to try the wiki out. I was delighted to set up the first group with their own branch of the wiki tree, and look forward to getting their feedback.
In a dumbed-down form appropriate for non-geeks, Wikis have great potential to be a key knowledge-management solution for a lot of content management problems in an organization. I think with Wiclear I’ve set up a foundation that won’t scare people away without even giving it a try, and that, in my organization, would be called a victory!