REQUIREMENTS

1. Easily support the IMAP extensions supported by Courier:
   • children
   • namespace
   • thread=orderedsubject
   • thread=references
   • sort
   • quota
   • auth=plain
2. Easily support all of IMAP's simple operations
   • auth
   • select
   • fetch
   • noop
   • etc
3. Be mountable by WebConnects
4. Offer an easy upgrade path for the existing mail store.

EXPLORATION OF A RESTFUL MAIL INTERFACE:

Base IMAP commands

The URIs -- Scheme, Authority, and Path

A restful interface is built around URI's. So, what are the URI's?

Well, I think that they'll look like this:

http://<server>/<user>/<mbox>/<mbox>/<mbox>?<operation>#operation-options

So, for example, if user dev2001@simdesk.com wants to access his mail in the folder INBOX on server maildev1, he would use a scheme, authority, and path of

  http://maildev1/dev2001@simdesk.com/INBOX

The question is what does the "query" part of the URI contain? Well, that would depend on the IMAP operation. So, what are the IMAP operations?

IMAP-Specific Commands, Commands from IMAP's Authenticated State

Well, beyond the IMAP-specific stuff (CAPABILITY, NOOP, LOGOUT, AUTHENTICATE, LOGIN, and STARTTLS) we have SELECT, EXAMINE, CREATE, DELETE, RENAME, SUBSCRIBE, UNSUBSCRIBE, LIST, LSUB, STATUS, APPEND, CHECK, CLOSE, EXPUNGE, SEARCH, FETCH, STORE, and COPY. (Well, there's also the "UID" command, that really just modifies the COPY, FETCH, STORE, or SEARCH operation that needs to follow it.)

Okay, I can see that SELECT and EXAMINE really just set the path part of the URI that the operations work in, so they really don't have a consequence as a main effect of their execution. However, they do return some data to the user that must be pulled from the correct place in the mail store. So, where does that data come from? Well, what data is there? The FLAGS value, the PERMANENTFLAGS value, the EXISTS value, the RECENT value. the UIDNEXT value, the UIDVALIDITY value, and the UNSEEN value.

Well, suppose I do something like this: To get all the mailbox meta-data, do something like a GET of:

  http://maildev1/dev2001@simdesk.com/INBOX?metadata

or to get selected bits of it, do something like a GET of:

  http://maildev1/dev2001@simdesk.com/INBOX?metadata#EXISTS,RECENT,UNSEEN

The response will include the data which was requested. A response to the whole data set might look like this:

  <metadata>
    <exists>23</exists>
    <recent>5</recent>
    <unseen>12</unseen>
    <permanentflag>\Answered</permanentflag>
    <permanentflag>\Flagged</permanentflag>
    <permanentflag>\Deleted</permanentflag>
    <permanentflag>\Seen</permanentflag>
    <permanentflag>\Draft</permanentflag>
    <permanentflag>\*</permanentflag>
    <flag>\Answered</flag>
    <flag>\Flagged</flag>
    <flag>\Deleted</flag>
    <flag>\Seen</flag>
    <flag>\Draft</flag>
    <uidnext>12345</uidnext>
    <uidvalidity>2</uidvalidity>
  </metadata>

So, that gets "SELECT" and "EXAMINE", which are really the same thing, from the back-end's standpoint.

So, what's next? Well, there's CREATE, DELETE, and RENAME. I can do the first two with the HTTP "PUT" and "DELETE" operations. How do I do a rename? Well, "POST" looks good.

So, suppose I want to rename "INBOX/foo" to "INBOX/bar". I would do this with a POST operation that had a URL like this:

  http://maildev1/dev2001@simdesk.com/INBOX/foo?rename#/INBOX/bar

Now, about SUBSCRIBE and UNSUBSCRIBE? Well, since that's an update to the state of the mailbox, I can do things like POSTs of:

  http://maildev1/dev2001@simdesk.com/INBOX?subscribed#1

and

  http://maildev1/dev2001@simdesk.com/INBOX?subscribed#0

LIST and LSUB look to be similar, but due to the strangenesses associated with LSUB, they aren't, really. I think what I want to do is use similar URL's, but with slightly different semantics. With LIST, I think I only want to do the folders immediately under the current folder. However, with LSUB, there is a requirement that folders be listed even if you aren't subscribed to them if there is a subscribed folder underneath them somewhere. So, I think I want to list ALL of the subscribed folders, and let the client sort out which ones it wants to see.

So, the first would be handled using a GET of this URL:

  http://maildev1/dev2001@simdesk.com/INBOX?folders

This might result in a response that looks like this:

  <folders>
    <folder name="INBOX/Drafts">
      <attribute>\Noinferiors</attribute>
    </folder>
    <folder name="INBOX/Junk">
      <attribute>\Noselect</attribute>
    </folder>
    <folder name="INBOX/Saved">
      <attribute>\Marked</attribute>
    </folder>
    <folder name="INBOX/Sent Items">
      <attribute>\Unmarked</attribute>
    </folder>
    <folder name="INBOX/Trash">
    </folder>
  </folders>  

The second would be handled using a GET of this URL:

  http://maildev1/dev2001@simdesk.com/INBOX?subfolders

The resulting data would look the same as the example given above.

So, now "STATUS". This command is used to return the data that you'd get from a "SELECT" or "EXAMINE", but without changing which folder the client is currently reading messages from, so it's already handled.

The "APPEND" command is used to add a message to a mailbox. Obviously, (or maybe not that obviously--I just spent an hour trying to figure out whether this should be a "POST" or a "PUT" and decided it needed to be "POST" because the action it operates on is the mailbox, not a client-specified item in the mailbox) this would be a "POST" to the mail box.

So, the action is "POST" and the URL to append a message to the INBOX is

  http://maildev1/dev2001@simdesk.com/INBOX

and the body of the request contains the message to be appended.

Commands from IMAP's Selected State

The "CHECK" command, and the closely related "NOOP" command are mostly used by clients to poll for new messages. The primary difference is that "CHECK" can take a lot longer to execute than "NOOP" for some servers if (for example) the server needs to flush a message box's buffers to disk. The "CHECK" command does that, the "NOOP" command doesn't.

I don't think I want to expose that level of detail about the implementation through this interface, so I'm not going to explicitly support it. "CHECK", like "NOOP" is not considered to have a URL directly associated with it. Instead, the statuses that the a client might expect to get as the result of a "NOOP" or "CHECK" command would be done with a GET of the metadata. The only problem with this approach is the requirement that the client be given a list of the MSN's of deleted messages at appropriate times, and the execution of a "NOOP" command or a "CHECK" command is one of those appropriate times. How do I generate those lists? Well, the way that I've come up with is to use a kind of serial number that specifies a revision of the mail store. This serial number would be updated at strategic points during the operation of the server. Details are in with "EXPUNGE" command.

The "CLOSE" command returns the session to the authenticated state, and has the side-effect of expunging all of the messages marked for deletion without sending a list of the MSN's of those messages to the client. That makes it's handling similar to the handling of EXPUNGE, without the list of deleted MSN's.

The "EXPUNGE" command deletes all messages that have the delete flag set, and it returns a list of the MSN's deleted. Logically, those are two different steps and would require operations on two different URL's to complete. The first URL instructs the system to purge all messages that have been marked for deletion. It's a POST of

  http://maildev1/dev2001@simdesk.com/INBOX?purge

The second URL retrieves the list of MSN's deleted since some previous time. As discussed in the paragraphs on "CHECK", I have to define a revision serial number to allow this to work because REST does not have any concept of session while IMAP is very session-oriented. So, presuming that the serial number is given in serial, that data would be returned from a GET of this URL:

http://maildev1/dev2001@simdesk.com/INBOX?last_deleted# serial

That data would look something like this

<deleted>
  <serial>123</serial>
  <message><msn>1</msn><uid>47</uid></message>
  <message><msn>1</msn><uid>48</uid></message>
</deleted>

Note that each message that has been purged is speciifed both by its msn and its uid.

Before continuing to specify the URLs associated with IMAP commands, it is helpful to specify the RESTful equivalents to four different IMAP structures: The sequence set, the searching criteria, the message data item names (for fetch), and the flag list. All of these will appear in the query section of the URI, and they will all be name-value pairs.

A sequence set will be either "uid=" or "msn=" depending on whether the sequence represents a set of unique identifiers or message sequence numbers, followed by an encoded version of an IMAP sequence set. IMAP sequence sets are a series of numbers or asterisks separated by commas and colons. All commas would be replaced by "%2c" and all colons would be replaced by "%3a". So, the IMAP sequence set of:

1,5:*

which represents the messages 1, and 5 to the last message, inclusively, would be converted to

1%2c5%3a*

The search criteria will simply be the IMAP search criteria, as specified by RFC 3501, encoded appropriately for a URL.

The message data item names will simply be the IMAP search criteria, as specified by RFC 3501, encoded appropriately for a URL.

In the view that I am taking of this system, the "SEARCH" and "FETCH" commands are considered to have similar purposes. To retrieve messages, or information about messages, one does a GET with a search attribute that gives the criteria used to select messages and one includes a fetch criteria to determine which information one expects back. The IMAP "SEARCH" command is quite complex because it needs to provide for considerable flexibility in those searches that can be submitted from the client to the server. In order to prevent having to invent a Simdesk-specific mechanism for specifying the search criteria, the search criteria will simply be the IMAP search criteria, as specified by RFC 3501, encoded appropriately for a URL, with one exception. The IMAP RFC specifies that a set of messages may be specified by MSN by simply including that sequence set in the search criteria. In a restful mail store, a set of MSN's must have "MSN" prepended to it.

The fetch attribute specifies what parts of the message or information about the message to bring back as a list of "message data item names". With one exception, the "message data item names" will be specified as the argument to a "fetch" parameter, using the format specified in RFC 3501, and encoded in a fashion appropriate for URIs. The exception is that a "MSN" data item, which simply returns the message sequence number of the message.

An equivalent to an IMAP "SEARCH" command might be a GET of this URI:

  http://maildev1/dev2001@simdesk.com/INBOX?search#FLAGGED%20SINCE%201-Feb-1994%20NOT%20FROM%20%22Smith%22&fetch=MSN

An equivalent to an IMAP "FETCH" command might be a GET Of this URI:

  http://maildev1/dev2001@simdesk.com/INBOX?fetch#RFC822&search=msn%201%3a*

Of course, it one could combine the search and fetch functionality to bring back message data from those messages that satisfy certain criteria. For example, to bring back the SUBJECT of all the unread messages, one might send a GET that looks like this:

  http://maildev1/dev2001@simdesk.com/INBOX?fetch#BODY%5bHEADER.FIELDS%20%28SUBJECT%29%5d&search=UNSEEN

The response to the fetch will be an XML document that includes all of the information, grouped per message, with different tags delimiting the different data items in the fetch parameter.

The "STORE" command accepts a list of UIDs or MSNs and updates the flag sets of those messages. The URI to PUT will include a "uid" or "msn" parameter, whose argument is the sequence set of messages to update, in the format specified above. An optional "silent" parameter will determine whether or not the new value of the flags are returned. The type of update to do will be determined by the command parameter, which is one of "store", "add", or "remove". The argument to the command parameter will be the flag or flags to perform the action on. This list is as specified for flag-list in RFC 3501 and will be encoded appropriately for URIs. A STORE uri that updates the flags for all messages in the mailbox to include the \deleted flag might look like this:

  http://maildev1/dev2001@simdesk.com/INBOX?silent&add#%5cdeleted&msn=1%3a*

The "COPY" command accepts a list of UIDs or MSNs and copies them to a specified mailbox. The URI to PUT will include a "uid" or "msn" parameter, whose argument is the sequence set of messages to copy, in the format specified above. A "copy" parameter will give the path to the destination mail box. The URI for a "COPY" command to save all of the messages in the INBOX to the INBOX/saved/2005q1 folder might look like this:

  http://maildev1/dev2001@simdesk.com/INBOX?copy#/INBOX/saved/2005q1&msn=1%3a*

IMAP Extensions supported by Courier IMAP

The Child Extension

The child extension adds "\HasChildren" and "\HasNoChildren" attributes to the responses to a LIST command. That implies that the only change required to support this feature would be the inclusion of those attributes in the appropriate responses.

The Namespace Extension

The namespace extension is something that is intended to support differing implementations for IMAP servers. As such, it doesn't really apply to a new RESTful interface to a mail store.

Threading and Sorting

The IMAP Thread and Sort commands are thought of as variations of the search command. That is, they both search through a message folder searching for matching messages based upon standard search criteria. However, they return their results in an order that reflects either the message threading or the order implied by specified sorting instructions. Since we're treating searching and fetching as different aspects of the same operation, all I'm going to do is to add "thread" and "sort" parameters to the GET. Further, "first" and "max" parameters are added so that a client can request a displayable subset of the message data being fetched.

Quota Support

Quota used is viewed as metadata about the particular mail folder. Therefore, retrieving mail quota data would be done using a fetch of a mail folder's metadata.

WebConnects Issues

It probably would be wise to define a mechanism to retrieve data in a JSON format, for easy use at the browser.

Upgrade Path from the Existing Mail Store

-- JonathanGuthrie - 06 Feb 2007

I have basically gone through and replaced most of the equals signs with hashes because it seems to fit the logic better.

-- JonathanGuthrie - 25 Apr 2007

-- JonathanGuthrie - 21 May 2008