FOREST: An Interacting Object Web

Chapter

Abstract

FOREST satisfies the need for objects to easily interact across the network in a RESTful way – without calling methods on each other. To do this, it asks you to set your objects up in an Observer Pattern relationship. Or, in particular, a “Functional Observer Pattern”, where an object’s state is set as a Function of its current state plus the state of other objects it Observes through links. This observation occurs through either pull or push of linked object state. Such a programming model maps directly to RESTful distribution over HTTP, using GET for pull and POST for push of object state, in both directions between interacting servers. Objects are published into a global interacting object Web. This distributed object architecture is declarative in nature, and thus very expressive, as well as being naturally concurrent.

FOREST satisfies the need for objects to easily interact across the network in a RESTful way without calling methods on each other. To do this, it asks you to set your objects up in an Observer Pattern relationship. Or, in particular, a “Functional Observer Pattern”.

FOREST is an acronym for “Functional Observer REST”. Once objects are in a Functional Observer relationship, RESTful distribution or integration becomes very simple, as there is essentially a one-to-one mapping from the Functional Observer model to REST’s Hypermedia Constraint.

FOREST is thus a distributed object pattern distinguished by its style of object interaction. It describes a Web of interlinked, interacting, interdependent object resources, hosted across multiple applications or servers.

Functional Observer means that these objects interact by setting their next state as a Function of their current state plus the states of other objects Observed near them in the Web and on which they depend. Observation may occur either by pull or by push of object state.

This observation is enabled across the network by an HTTP layer using just GET and POST, in a “symmetric REST” style, conforming to the constraints of REST (Fielding 2000). A server may become a client in order either to GET or pull a remote object resource for its own dependent object resource, or to POST or push its own object to a remote dependent object. Another way of looking at this is that a client becomes a first-class server to publish its own object resources.

Although an unusual object interaction pattern at the programming language level, the Functional Observer model is actually very easy to program due to its declarative, functional, reactive and asynchronous programming style. Functional Observer objects are programmed independently as “masters of their own evolution”. It is similar to approaches found in the Clojure and Erlang programming models, and may be readily implemented over the asynchronous Node.js framework, as well as mainstream languages such as Java.

This object independence makes Functional Observer object interactions easy to distribute, both across application partitions inheriting the interoperability, evolvability and scalability benefits of REST and across multicore, without the usual concerns around threads and locks.

Functional Observer Pattern

In the Functional Observer Pattern, an object sets its own next state as a Function of the object states it Observes in itself and through its links at any time (Fig. 7.1):
Fig. 7.1

Functional Observer

Functional Observer objects do not interact by calling methods, they simply exchange public state with each other. An object is master of its own evolution. When it needs to set its own state for example, when observed itself, or after being notified of an update to a linked object it is observing it looks at its own content state and the state of all other objects around it local and remote that are visible through links (or links to links via a chain of objects) and on which it depends. Then it moves its own state forward, guided by functions describing the application business rules or domain-level constraints. That new state is then notified on in turn to other dependents or observers.

An important constraint in Functional Observer is that observed objects do not get to know what other objects are observing them. However, an object “A” can make itself known to another object “B” that does not yet have any link access to it, by pushing itself to “B” unasked. Object “B” can then set itself up as an observer of object “A” if it then adds a link to “A” and carries on scanning it.

Objects have unique ids, perhaps UUIDs or GUIDs, by which they are fetchable or observable from an object cache. Their content the data that has a state at any time is a structure of common programming language elements: strings, numbers, lists and hashes, with links to other objects via their unique ids.

Functional Observer and Related Styles

It is beyond the scope of this chapter to discuss the history and origins of the Functional Observer model and its place within the broad scope of programming models, patterns, styles and paradigms. We will simply point out some similarities and mention some languages that support implementation.

The Functional Observer model is somewhat related to Dataflow, Reactive and Functional Programming styles, although it can be readily implemented in mainstream languages. It has some similarities to, and perhaps may be implemented by, Clojure’s Agents and Watchers (Hickey 2009). Similarly, implementations of the Actor Model, including Erlang (Armstrong et al. 1996), may support the Functional Observer Pattern. A useful, but more basic, infrastructure for implementation is provided by the “Node.js” asynchronous, server-side Javascript framework (Dahl 2009).

Functional Observer objects interact in a declarative, reactive and asynchronous way with other objects. A Functional Observer object never directly tells another what to do by calling methods on it. (It may strongly imply a preference directed at another, however, which can have a similar effect, but most often that style of interaction is not needed.) This switch from imperative to declarative thereby causes a corresponding inversion of the Object-Oriented mantra, “Tell don’t Ask” (Cragg 2009).

Implementation of Functional Observer

The implementation of Functional Observer is very simple. Normally, a callback on an object will be set up, triggered by a pushed incoming state change on a previously observed, linked object; or by object creation or re-cacheing. Then the object will go around scanning links to observe, and pulling state on which it depends, to decide what its current state should be, or what its reaction will be to an incoming new state. When it changes itself, its state gets pushed on in turn to the objects that have previously depended on it.

The act of scanning an object through a link (or link chain) sets up the observation of that object’s next update. Observation can occur either by pull the observer reads the state of the linked target object when needed or push the target pushes its state to the observer when it changes. This can thus support both lazy and eager programming models.

An object observes the links it visits in this process, and conversely, if it no longer visits a link, it stops observing it. An object that falls out of the cache through lack of use may also stem observation of its dependencies. Some objects may allow themselves to be observed at any time, some only when they have re-evaluated themselves by observing their own dependencies.

A difference between Functional Observer and the traditional Observer Pattern is in the way events are notified. Functional Observer operates asynchronously, rather than pushing out the notifications on a single thread. Hence notifications are queued, and thus may be handled by another thread or another process on a remote machine. Reads cacheing objects in and GETs of remote objects are also asynchronous. This asynchronicity means that an object needs the option of deciding to block or refuse observers when its state is not ready; when it is waiting for a dependent state of its own to ensure it is up-to-date.

Functional Observer REST

The familiar model of method-calling that is used for object interactions in common programming languages is, of course, not resonant with distribution in the REST style. We identify this method-invocation interaction style as “imperative”. RPC is an example of this style when attempted over the network.

Functional Observer’s declarative object interaction style, on the other hand, is resonant with REST, and maps straightforwardly onto RESTful use of HTTP; and in particular onto the Hypermedia Constraint, as we will show. Correct REST distribution or integration has often been seen as a tricky art, so there is a huge advantage in having a simple and powerful programming model where RESTfulness almost “drops out” (Fig. 7.2).
Fig. 7.2

Functional Observer REST

FOREST’s Functional Observer object interactions map onto HTTP bidirectionally: to pull linked object resource state using GET, and to push state using POST, in the style of a “reverse GET”. As a result, using POST this way is an idempotent operation. Object content is serialised into an appropriate Media Type.

POST notifications occur by the sender knowingly pushing itself to a target object: observation and awareness of updates is manifest onto HTTP as either conditional GET and polling or POST for this deliberate push. As objects do not get to know their observers, a GET need not carry information about which object is asking.

These state transfers by GET or POST are all that is needed to allow each object resource to meet its domain logic dependencies on other linked objects. HTTP is only used, in this pushpull mode, for the domain-independent exchange of state over the network; all domain-level conversations occur only up at the level of the object state itself, within the Functional Observer Pattern interactions. Using the HTTP layer for generic state transfer, supporting the interactions in the object layer, gives clear separation of concerns.

One consequence of Functional Observer’s declarative interaction style mapped into HTTP in this way is that the two, less-used, HTTP verbs that often come up in REST patterns PUT and DELETE have no place. There is nowhere in FOREST where it makes sense to imperatively or directly attempt to replace or delete someone else’s object. HTTP is not used by a client application to try and tell a server application what to do; it is just used for the bi-directional state exchange by pull and push, GET and POST.

It is via this network distribution that we move from just Functional Observer to FOREST, which as we will show, naturally follows the constraints of REST.

FOREST Foreign Exchange Trading Example

This will all become much clearer with a real example. Rather than dig into how a programmer sees the Functional Observer Pattern play out, with all the details of implementation languages and programming styles, it will be enough simply to watch what happens “on the wire” in HTTP messages, as some objects held on different servers interact with one another.

This will bring out the detail of both the Functional Observer interactions and the application of the constraints of REST in FOREST. The second part of this chapter will then show in detail how FOREST meets the Hypermedia Constraint, which is a little more complex than can be explained by way of this example. The Stateless Constraint is also left to be discussed in that context.

We’ll demonstrate with a Foreign Exchange order fulfilment scenario. This will have an Order server holding Orders and a Fulfilment server holding fulfilment status as Tickets. The Order server can also hold Payments. Hence, in the Functional Observer style required by FOREST, Orders may observe Tickets and Tickets may observe Orders and Payments.

First, an as-yet-unseen Order in the Order server (fx-orders.com) observes, and thereby fetches, a known, top-level “Dealer” object from the Fulfilment server (fx-broker.com):

GET /dealer123 HTTP/1.1 Host: fx-broker.com HTTP/1.1 200 OK Etag: "313" Cache-Control: max-age=10 Content-Type: application/json { "tags": [ "fx", "dealer" ], "tickets": [ "http://fx-broker.com/tick110", "http://fx-broker.com/tick109" ] }

Just this short, simple interaction gives us very much to discuss about the application of REST’s constraints in FOREST.

Client–Server; Layered; Cache

So: which is the client and which is the server? In this scenario, as we will see, the Order server can be a client of the Fulfilment server and vice-versa.

In general, servers have responsibilities at a domain or business level, which are not always easy to partition into simply either client and server at the network level.

But if we look over Fielding’s thesis, we can see that “separation of concerns is the principle behind the clientserver constraints” and “the separation allows the components to evolve independently, thus supporting the Internet-scale requirement of multiple organizational domains.”

Thus having servers that are also clients is fully compatible with the intent of the ClientServer Constraint as long as this separation of concerns exists in our Order and Fulfilment servers. Also, using HTTP means the clientserver interaction style is baked in at the level below.

Finally, the REST definition states as an advantage of the uniform interface that servers such as search engine crawlers can act as clients. We will refer back to this below, as it is one of the closest ways the Web comes to our fulfilment scenario.

We’d like to be able to use proxy-caches and other intermediaries, via the Layered Constraint, which we can do by using HTTP correctly. It actually ties in nicely to our more two-way version of the ClientServer Constraint, since proxies are also both clients and servers. Further, our client-like servers can act as intermediaries when isolating concerns in more complex scenarios than this fulfilment one.

We have some headers that will help us meet the Cache Constraint of REST, in the Etag and max-age. We should be able to cache Orders in the Fulfilment server, and Tickets in the Order server, as well as in any proxy-caches in between.

Identification

In FOREST, we only have real object resources of the host’s implementation language, not resources that are in any way abstract. They will have unique ids in that internal object world. Hence, they can always respond to GET requests on URLs that map to their ids with their object representations or serialisations.

Now, we’ve given the Dealer URL a simple, readable name. But in real life, that URL is more likely to look something like: “http://fx-broker.com/xyz/312a–5990bf4–34cd007.json with an internal GUID or UUID appearing in the URL.

Notice that we’ve used the more common acronym “URL” here, instead of the more generic or “correct” one, “URI”, that is most often used in REST discussions. The definition of “URI” includes potentially unfetchable referents, identified by strings intended for human consumption, such as URNs [or URLs with fragment identifiers (Berners-Lee et al. 2010)]. Our URLs are always fetchable and always opaque, at least in principle. “URL” has a clear meaning that is exactly what we want (Mealling and Denenberg 2002; Berners-Lee et al. 2005) and which allows us to be very precise when discussing the REST Identification Constraint, thus avoiding any philosophical or Semantic Web entanglements. Anything semantic has to appear in an object’s content, not in the links between objects.

If including an object UUID or GUID in its URL, the “Identifier” aspect of the URL is essentially given by that element. Then the domain or host part of the URL is the “locator” aspect for where that uniquely- and universally-identified object can always be found. Indeed, any other characters of a URL are then essentially redundant, including “.json” or “.xml”, other path elements, etc.

Further, given that the UUID or GUID is universally or globally unique, it is certain that a representation or copy of the object referred to could also be found cached in any one of a number of other hosts perhaps keyed by the unique object id alone. It is the ability to reliably find or locate an object representation by a unique identifier that is the essential characteristic we need.

Self-Descriptive Media Types

We are using JSON for our serialisation of objects into their representations the Content-Type is application/json as it is simple and clear, and a better match for the kind of data we prefer to program our objects with.

This also allows us to conform to REST’s Self-Descriptive Constraint for Media Types as this is a common, standardised format, widely understood by available libraries and frameworks.

The purpose of the Content-Type header at the HTTP layer is to give broad direction to choose processing modules within which most of the actual content interpretation happens, once that module starts reading inside the content itself. In the content layer, there will always be an understanding between client and server that goes beyond the Media Type and into business or domain interactions.

For example, there is a channel of communication set up between an author of plum jam recipes and their plum-picking readers, carried through HTML. On an e-commerce Web site selling plum jam, HTML is exchanged such that the site can have an e-commerce interaction with the jam-consuming end-user. That exchange is carried by, or executed through, the base HTML Media Type.

There is not one Media Type for recipes, and another for buying jam! Minting new Media Types, maybe one or more Media Types per application perhaps as “vnd. ∗ ” or “ ∗  + xml” types compromises self-descriptiveness.

At the HTTP level on the Web, the Content-Type is a very broad and coarse label, which has the great advantage of self-descriptiveness: REST requires that we have few Media Types understood as widely as possible; including by installed libraries and applications. Our guidance on Media Types is to follow the crowd even, indeed, in preference to following a standard.

So we should use a very common base type, called out in HTTP’s Content-Type, and have all the rich domain-level semantics up inside the content, driving the Functional Observer interactions. Hence, we could re-use XML or XML-based types such as XHTML1 or XHTML5 or Atom and AtomPub to serialise objects. More appropriately and simply for our data-oriented applications, we could output JSON representations of the internal object resources. Or we could use content negotiation to choose the serialisation.

But JSON is very bare and basic compared to XHTML. As we gain experience, we could aim to settle on a single, common, standardised sub-Media Type of JSON, with various domain-specific formats or schemas inside that type such as our Orders, Payments and Tickets. Perhaps we may call it “application/forest + json”, following the “ + xml” convention.

All this depends to some extent on how you view the strictness of the Self-Descriptive Constraint with respect to Media Types, which Fielding has called out as something of a frustratingly variable constraint in REST (Fielding 2006).

Self-Descriptiveness Inside the Content

As a publisher, unless you make a reasonable attempt to re-use all or part of any formats or schemas within XML, XHTML or JSON that are in wide use or being standardised, including Microformats, etc., you are still reducing your conformance to REST’s Self-Descriptiveness Constraint. You still need to be aware of the trade-off of precision versus re-use that this implies.

As long as the base Media Type is conformed to, this re-use can be in the form of basic instantiation, sub-classing or extending. Also, the base type can be used as a carrier for another type, in the way Microformats, RDFa or Microdata are carried by XHTML1/5 and content is carried by Atom. It may also take the form of just re-using parts of existing standards.

On the client side, however, different recipients of such data see things their own way. Class is in the eye of the beholder, not synchronised through shared libraries. This allows decoupling and independent evolution.

The way a Media Type is interpreted or processed is entirely up to the client. The server is signalling what it thinks this content is in general terms, but the client can and will interpret that Media Type however it likes given its current state and goals. Whenever you fetch a JSON- or an XML-based object, the schema above that base  i.e., where you expect to find the data you were looking for will always depend on the URL you fetched it from, the context around that link, and the use you expect to make of it.

In the spirit of Postel, decoupling is enhanced through a server’s respect of standards in its published data, along with a client’s tolerance of the incoming type just taking what it understands and ignoring anything else, perhaps by using XPath or an equivalent approach in JSON.

In our JSON content, we’ve chosen a tagging system for creating class-like subtypes of the base Media Type. The “fx” and “dealer” tags give us flexible, loose typing which sets our expectations about how we interact with this object, from reading some documentation and conformance tests. The aim would be to settle on a number of common JSON schemas or grammars, using such conventions, within our “application/forest + json” Media Type.

Self-Descriptive Methods, Headers and Response Codes

REST does not talk much about the subject of methods, except to say that they should form a uniform interface and that they should be well-known.

As we’ve already described, our use of GET and POST is entirely for state transfer in each direction initiated by either side. This is pretty conformant to Self-Descriptiveness, as the Web works much like that. The idempotency of POST in FOREST can be signalled by use of our “application/forest + json” Media Type.

Using PUT and DELETE can actually fail us in self-description of messages, since much HTTP code in the wild does not handle them. However, of course, they are not necessary for the Functional Observer Pattern.

Correct use of common HTTP headers and response codes gives proxies and clients a chance to cache content and to know when things have gone wrong in the state transfer. We expect to see the usual response codes: “200 OK”, “303 See Other”, “304 Not Modified”, “404 Not Found”, and others.

Further helping self-descriptiveness and separation of concerns, these headers and return codes should be given as closely as possible their common meanings regarding just the state transfer of representations. In FOREST, we do not modify their meaning in any way on a domain-specific basis. Thus “200 OK” means the state transfer was successful even if that state indicates an “error mode” at domain level, in a Functional Observer object exchange.

Hypermedia and Hyperdata

Although we will revisit the Hypermedia Constraint later, we still need some hypermedia to constrain! In fact, in FOREST, our hypermedia is more in the nature of “hyperdata”. Hyperdata means having our data linked up into an object Web or a graph of objects.

Here, in the response content representing the Dealer object, we see that the JSON object returned has URLs inside JSON strings. It is currently dealing with a number of Tickets, listed within it.

These links are created in a fairly obvious way. Presumably our Dealer object links internally to these Ticket objects. So when serialised into a Dealer representation, those links can simply be converted to URLs containing the Ticket objects’ unique ids. As we’ll see below, the Ticket object will also point by URL to its corresponding Order object over on the Order server which points back thus completing a cross-server, hyperdata object Web.

Now, declaring that a URL found in a JSON string is hyperdata is hardly something worthy of an RFC, but that would be part of any “application/forest + json” JSON hyperdata standard. Similarly, you may prefer XHTML1/5 to XML, as it comes with a built-in link semantics, where XML would need to be enhanced with XLink.

Links in FOREST only represent such declarative data structuring, and therefore always point to fetchable object resources. Link relations may be used in XHTML or XML as part of the serialisation of the structure of an object linked to another, but the entire surrounding context of a link, plus the goals and intents of the observer, give the link semantics, not just a single relation tag. Finally, a link may sometimes just have to be fetched before you can really know what it meant!

As long as all of your interacting servers are able to discover and use the links they need that are in the data, guided by surrounding context, that is all that matters. Thus, we can see and traverse links in our hyperdata. We have a distributed graph of objects; a hyperdata object Web, both created and consumed by the applications being integrated.

Back to the Example…

After that very long detour, triggered by a simple GET, things should go a little quicker now in our Foreign Exchange example. Here is that GET by the Order observing the Dealer again:

GET /dealer123 HTTP/1.1 Host: fx-broker.com HTTP/1.1 200 OK Etag: "313" Cache-Control: max-age=10 Content-Type: application/json { "tags": [ "fx", "dealer" ], "tickets": [ "http://fx-broker.com/tick110", "http://fx-broker.com/tick109" ] }

An Order object is coded to understand Dealer objects. It knows that this is the entry or starting point of its transaction, as long as it can make that Dealer notice. So it sets itself up to trigger a POST of its own state to that Dealer an unsolicited push or notify in Functional Observer terms, which can make the Dealer an observer of the Order as long as it is interested:

POST /dealer123 HTTP/1.1 Host: fx-broker.com Content-Type: application/json { "%url": "http://fx-orders.com/ordr321", "tags": [ "fx", "order" ], "params": [ "usd/jpy", "buylim", 81.7, 500.00 ], "dealer": "http://fx-broker.com/dealer123" }

This JSON Order object includes its own URL at the Order server, with a preceding “%” on the tag as a convention indicating a metadata field. Such fields are candidates for including in the HTTP headers; the URL could be included as a “Content-Location” header, for example. This depends on your tolerance of non-standard HTTP header usage.

The appearance of this URL reflects firstly, that this is a first-class object available on the Order server whose state is being pushed, and secondly, that this POST is idempotent, as are all FOREST POSTs, because reporting the latest or current state is an idempotent operation. This POST is a state declaration; it is idempotent in intent the Order is just telling the Dealer of its current state implying that it is ready to be told about a corresponding Ticket. POST is state transfer in FOREST, not an event, message, action or command. The role of POST will be discussed further below.

The “dealer” URL in the Order object indicates that this is an Order for this Dealer, no other. The Order has to stand independently, declaratively; potentially fetched by others using GET. There will be many such links in Functional Observer, since interactions within and through an object Web is fundamentally how it all works.

The order itself is for a USD/JPY exchange “buylim” is short for “buy limit” which means buy when the price drops far enough below 81.7 here.

On the Fulfilment server side, there is a constraint or rule on the Dealer that ensures it interprets this incoming POST as a state declaration, not a command that could result in multiple Tickets: “if I observe an Order pointing to me that does not have a link to a Ticket, I must make sure I have a Ticket somewhere in my list that points to this Order”. Such rules work off state, not state change, so can be re-run repeatedly.

So, within the Fulfilment server, the Dealer object sees the incoming Order object and creates a new Ticket object, adding that object to its list.

Then it is up to the Ticket to carry on the conversation with the Order. The Ticket object is now a “live” object, responsible for its own evolution. Once the Ticket has sorted itself out, it needs to tell the Order object about itself.

A POST response is something of an open channel for now returning state. The HTTP RFC is rather vague about what is returned from a POST. In theory, it would be possible to always return a “204 No Content” empty response to all POSTs, and then POST back any follow-up updates separately. But that would be a waste it is more efficient to use this opportunity to update the POSTing server immediately with any new state that has occurred as a result of the incoming POST.

So, the most likely response to a POST, and usually the most useful, is for it to act exactly as if it were a GET on the POST URL target. It returns a 200 code and sets the Content-Location header to its own URL, to indicate that this response is intended to be seen as equivalent to a GET (Fielding et al. 2011). This gives the target a chance to immediately and efficiently return in the response any new state that was triggered by the incoming object POST.

Here is such a response that we could get to the POST on the Dealer, showing our new Ticket object in its list:

But in this case, it is not telling us much. Alternatively, we could return another object that is considered a dependency of that incoming object, either through the same 200/Content-Location approach, or through a 303 redirect, also containing the object itself to save a round-trip. In this example, the Ticket object itself would actually be more useful to us, instead of just the Dealer pointing to it:

HTTP/1.1 303 See Other Location: http://fx-broker.com/tick111} :

or:

HTTP/1.1 200 OK Content-Location: http://fx-broker.com/tick111} Etag: "1" Cache-Control: max-age=10 Content-Type: application/json { "tags": [ "fx", "ticket" ], "order": "http://fx-orders.com/ordr321", "params": [ "usd/jpy", "buylim", 81.7, 500.00 ], "ask": 81.9, "status": "waiting" }

The Ticket object confirms the params, current asking price and status.

If for some reason the first approach were actually taken, the Ticket would have needed to push itself back at the Order, again by an unsolicited push:

POST /ordr321 HTTP/1.1 Host: fx-orders.com { "%url": "http://fx-broker.com/tick111", "tags": [ "fx", "ticket" ], "order": "http://fx-orders.com/ordr321", "params": [ "usd/jpy", "buylim", 81.7, 500.00 ], "ask": 81.9, "status": "waiting" } HTTP/1.1 204 No Content

[The Cache-Control and Content-Type headers will not be shown from now on, for brevity.]

These would be alternative mappings to the HTTP layer of the Functional Observer layer’s Ticket pushing itself to the Order.

As seen here, a POST can also return a “204 No Content” when there is no interesting new return state to report. In theory, a “202 Accepted” could be used instead of 200s or 204s, but that is not particularly meaningful in FOREST, since it is all asynchronous.

Finally, an object can indicate that it is not interested in, or no longer dependent on, a pushed/POSTed object, by returning a “403 Forbidden” status code (or a 405 if it is never interested in POSTs). Note that, although this may stem the POSTs at the HTTP level, the pushing object should not depend on these error status codes to decide what to do: it should only look at and depend on the content of the troublesome push target.

All POSTs can be retried if the response was not received, POST being idempotent in FOREST. As can all GETs, of course, including for polling. Notice that max-age gives our polling algorithm something to work on when calculating an optimum polling frequency. If this Ticket state did not get POSTed after a time period, then either the Order could be re-POSTed if the Order server timed out first, or the Ticket could be re-POSTed if the Fulfilment server noticed its POST failed. Note that max-age is set by the object itself. Also note that there will be both HTTP-level and domain- or business-level timeouts and retries.

Using just the two HTTP methods, GET and POST, and their corresponding headers and status codes and appropriate timeouts and retries, to manage the pull and push of object state between interdependent objects, leaves all the business-level or domain-specific object interactions separated up into the layer above in the content and its types and formats over or within the common Media Type. Which is also how the Web works.

Back to our example: the asking price is currently too high. But we are confident of a fall, so we’ve decided we’d like to invest more. The Order is updated, and pushes itself now directly at its own Ticket:

We are in luck, the market is still heading down. The Ticket is updated accordingly: its state is a function of the params in the order, combined with its internal access to the Foreign Exchange market.

Time passes. Anxiously, we poll to see if we missed anything because of a dropped POST:

GET /tick111 HTTP/1.1 Host: fx-broker.com If-None-Match: "2" HTTP/1.1 304 Not Modified Etag: "2"

No nothing is changed, the market is not moving.

Ah now the price has dropped and our Order is filled. We immediately get a notification pushed to the Order:

Now, while waiting for that we started to feel even more confident in our USD/JPY prediction, and wanted to bet on an even cheaper price. However, we now have a race condition:

Now the params do not match and it is too late, so this is flagged in the JSON “status” field. As explained above, this domain-level “error” condition is a Functional Observer object state exchange that still gets transferred at the HTTP level with a “200 OK” status.

Temporarily frustrated, we cancel the Order, with an update to its state:

No problem, the Ticket is marked as cancelled presumably the Dealer thinks that that was indeed a good price and will keep the purchase themselves. Again, we do not use DELETE because cancellation is a domain-level object interaction, and DELETE does not make sense when we are using the HTTP level for state transfer only what would you delete? Trying to delete anything would be an imperative call, and this is all about state declaration.

Actually, on second thoughts, we’ll take it at that price:

Not too late. Notice how this is all state-driven, not event-driven. All dependencies are state-dependencies. We are just applying business domain rules or constraints over dependent state. And in this case, as long as the back-end systems still allow the order to be filled at the stated conditions, we are on. There is no rigid state machine telling us we cannot go from cancelled back to ordered-and-filled. The business domain rules constrain relative state at any time, not a strict sequence of events.

Now to pay for the deal. We add a link to a new, locally-hosted Payment object:

Now the Ticket, that can see this new Order state, needs to traverse the “payment” link to see it, to determine its next state hopefully to “paid”. Hence there is no change just yet to the Ticket, so it returns a 204, then fetches the Payment:

GET /paym432 HTTP/1.1 Host: fx-orders.com HTTP/1.1 200 OK Etag: "1" { "tags": "payment", "invoice": "http://fx-broker.com/tick111", "order": "http://fx-orders.com/ordr321", "amount": 81600.00, "account": {.. } }

We could have POSTed this Payment to the Ticket instead, after POSTing the Order pointing at it. It is only shown this way as a reminder that the state transfer is essentially independent of which server initiates it.

If we POSTed, the idempotency of that FOREST state transfer would ensure multiple submissions were ignored, assuming the server co-operates. Again, there is probably a rule in there on Ticket that says: “If I’m aware that there is a Payment object pointing to me and I’m unpaid, take the payment (once) using that Payment object”.

Notice that this can be a generic, non-Foreign Exchange domain, payment format or type it does not say “fx” in the tags. And we can use normal Web security such as Auth headers, TLS, etc., to make this secure.

Now the Order sees the update to the Ticket: all done, all paid. Again, if the Payment were POSTed directly to the Ticket, this Ticket state would probably form the POST response instead.

The Hypermedia Constraint

That example showed how Functional Observer REST works, and how it meets all of the REST constraints, apart from two constraints not yet covered: the Hypermedia Constraint and the related Stateless Constraint.

It turns out that the Hypermedia Constraint maps right on to the Functional Observer programming model.

Stateless Constraint

Stateless means that the server holds no ongoing state between client requests to track or in any way drive or co-ordinate a succession of such client requests or anything else about the client, beyond offering responses on demand, containing links to more resources. Each request is seen as independent by the server at the HTTP level, and any state held across requests is only in the state of its own resources.

In FOREST, object resources may establish a relationship with “client-side” objects, but the interactions, inter-links and state are out in the open in the content, not hidden away at the HTTP interaction level, or in any client-specific session state. The HTTP layer of FOREST only handles the mechanical mapping of Functional Observer pulls and pushes between domain-level object resources into one-shot, independent GETs and POSTs.

So if the server is not holding state for the client, the client has to do all of its own co-ordination in the face of the server resources it sees. A client has its own “application state” to guide its working through the hypermedia Web before it. Indeed, hypermedia is the “engine” of this application state in REST the Hypermedia Constraint.

The Hypermedia Constraint thus works in hand with the Stateless Constraint, since server statelessness with respect to clients leaves its hypermedia as the only influence it has on client or application state.

Hypermedia Constraint

The phrasing of this constraint in Fielding’s thesis is “hypermedia as the engine of application state”, which he then goes on to elaborate: application state is the total state of the client, including current pages, open tabs, history, bookmarks, rendering of page, rendering of images, prefetching of links, etc. The state is stable once all requests are done with.

Hypermedia is the “engine” of this application state because the current state of a client application is directly dependent on the pages and images found by links, whether fetched automatically or by user action. Evolution of a client’s application state is a non-deterministic, heuristic, parallel and user-driven exploration of the server-side hypermedia landscape. So you could say “hypermedia and the user are engines of application state”. The hypermedia content and structure simply “is” and it is entirely up to a client, including the user, what it sees and where it goes to see it and when and in what order. That hypermedia structure does not imperatively dictate the evolution of application state, it declaratively guides or constrains it.

This even applies to the Web crawler, which will automatically evolve its “crawler application state” as it jumps from link to link. This application state is an index of pages it discovers. Now, when re-published as a search engine’s results pages, this application state becomes hypermedia itself. Hence, in this case, we have “hypermedia as the engine of hypermedia”!

The benefit of this constraint is that it encourages a declarative evolution of system state, rather than servers statefully walking clients through their imperative agendas. Servers send some hypermedia then forget the client even exists. Given a hypermedia type, the client can dynamically make its own choices without prior clientserver coupling.

The Hypermedia Constraint is also related to ClientServer in the sense of separation of concerns and independent components. Finally, both the Hypermedia Constraint and Self-Descriptive Media Types deliver loose coupling: clients are free to evolve both their own application states at run-time and their interpretation of incoming data independently of servers, and servers can similarly evolve their hypermedia within its type, or evolve the type itself, in a backwards-compatible way, independently of clients.

Applying the Hypermedia Constraint to the Fulfilment Scenario

The Order server provides hypermedia to the Fulfilment server, in the form of an Order which links to a Payment, so in that case, it is the Fulfilment server whose application state we want to be dependent on this Order server hypermedia. The Fulfilment server is akin to the search engine crawler in this way.

So what exactly is the “application state” in the Fulfilment server?

Consider what depends on the Order server’s hypermedia: the Fulfilment server’s Ticket, corresponding to the Order it has been given, is dependent on both that Order and its linked Payment. So the Fulfilment server’s Tickets are also its application state. There is no other state that is significant here. The Ticket can hold links to the other object resources that it is working on or cares about; Tickets hold links to Orders and Payments.

Driving Ticket lifecycle is the entire purpose of the Fulfilment server, just as rendering documents is the entire purpose of the browser, and collecting and indexing pages is the entire purpose of the search engine crawler.

And like the crawler through the search engine server, we can publish the application state we have evolved to the Tickets as more resources, more hypermedia.

For us in our fulfilment scenario, the Hypermedia Constraint quickly boils down to: the current state of a Ticket depends on the state of its Order and Payment. And, in the other direction, when the Order server is being the client, the Order depends on the Ticket, to see how it is being fulfilled and to make the payment when it is ready.

Which, of course, is another way of describing the Functional Observer Pattern.

Hyperdata as the Engine of Hyperdata

In other words, we meet this REST constraint in FOREST simply by following the Functional Observer model, ensuring that the states of the objects in our hyperdata graph published by our integrated applications are dependent on the states of other objects around them, discovered through links. HTTP is used as the pipework of that engine, to move object state around.

Or, to paraphrase for FOREST: “Hyperdata as the Engine of Hyperdata”. Which is exactly the Functional Observer model. Our use of Functional Observer, when distributed, gives us the Hypermedia Constraint, which, when distributing over HTTP, pretty much delivers all of the remaining REST constraints, too.

We have an interacting, interdependent, interlinked object Web. Each object in that Web is independent while also interdependent: the master of its own destiny, deciding how to evolve by itself, based upon the hyperdata context it finds itself within the context set by other, similarly independent yet interdependent objects.

Of course, some state has to be dependent on incoming events or external processes, rather than being entirely driven from internal object interactions, as in our example, where Tickets depend on the Foreign Exchange market. However, all internal interactions should work through the interdependencies of our interlinked objects in order for FOREST to conform to REST.

The Role of POST

The Hypermedia Constraint is largely GET-oriented. The Fulfilment server can GET the Order that a Ticket it hosts links to, or its linked Payment, from the Order server, because that Ticket depends upon them. The Order server may GET the Ticket, because the Order depends upon it.

However, in Functional Observer, we have push by POST as well as pull by GET. As we have seen, the Order server will also POST the Order directly to the Ticket on the Fulfilment server; and, indeed, that is what one would expect to be the normal flow of events when one server has something to say to the other. It is more efficient to push than to pull or poll when things are changing.

But how does such a POST of an object resource representation fit into the REST constraints? What are the responsibilities of POST in this scenario and in this interacting object Web interpretation of REST?

As already noted, REST has little to say on the actual methods we use. The general consensus in the REST community seems to be that POST can be used in pretty much any way you like, if you are not using its data-editing interpretation as “create a new entry in a collection”.

Since REST has little to say on the subject of the workings of POST, we could look at the HTTP spec, (Fielding et al. 1999), which currently says something about “subordinates”, alluding to the create-new role, but also has a catch-all “data-handling process” function, which is not especially useful to us. This is one of the areas that may be clarified by the HTTPbis working groups (Fielding et al. 2010).

So, we turn finally to the way the Web works, and it all actually becomes very simple, even to fit POST back into the REST terminology.

POST in REST Terms

On the Web, what we can POST, and to where, is set up for us with forms. Forms supply domain-specific annotation and structure, as well as a URL to POST to. Presenting a form and having the user fill it in is entirely within the realm of application state. That we got a form at all is an example of hypermedia being an engine of that state.

This application state then makes its way back to the hypermedia realm through the submit URL, and in any ongoing redirects and pages that depend on the submitted form.

Thus, in REST terms, POST is used when certain application states are reached, to send a little of that application state back into the hypermedia graph. That hypermedia may then change, in resource state or link structure, to drive our application state in a different direction. It is “hypermedia as the engine of an element of application state that then acts as an engine of hypermedia”!

So, in our scenario, where application state comprises first-class object resources, this becomes even more straightforward. The Order server knows, in the face of Ticket hypermedia or hyperdata, that it can POST an Order, as its submission of a part of its own application state, to the Ticket on its URL. Then the Ticket and its local hyperdata can change as a result of this POST; perhaps then driving the Order application state that depends on it in a different direction. Similarly, the Ticket can be POSTed into Order hyperdata, which may also react.

Logic Drives Push Between Interdependent Objects

The Order server knows that it can push the Order to the Ticket because it is part of the domain interaction specification for Order and Ticket objects the Order itself drives the push of its own state. This is the equivalent in automatic business or domain logic of a human driving an interaction, reading and filling in forms. The logic that determines the dependency of an Order on a Ticket is the same logic that determines that that Ticket is now likely to be interested in being POSTed the current Order state.

Indeed, that the Ticket may itself be dependent on that Order. From the Ticket’s point of view, as application state that depends on Order server hyperdata, it is continuing to meet its obligations under the REST Hypermedia Constraint to be dependent on the Order hyperdata, but instead of using GET to pull it, it is being pushed the Order, including its own URL for future GETs and perhaps with a hyperlink to the Payment.

It runs its own domain logic over the hyperdata graph visible in front of it, and this logic is indifferent to whether that hyperdata was pulled or pushed into view. Of course, once an object with its URL has been seen from a push, it is always possible to pull or poll on that URL, as long as it is saved as a link back.

Thus the initial POST is the key event once the target has received that, it takes over responsibility for its dependency on the incoming object. Subsequent POSTs are simply timely notifications of state change when the pushing object wants to take responsibility for that. Otherwise, the target can poll when it is interested.

Optimising POST

That concludes the description of the mapping between Functional Observer and REST’s Hypermedia Constraint over HTTP, via the mapping from pull and push observation to GET and POST. It is reassuring that this often tricky constraint of REST can be met easily by using the simple and powerful Functional Observer object programming and interaction pattern.

It is worth now exploring the role of POST as idempotent object push in FOREST, as there are some optimisations that can be made, knowing how it is used.

POST as Cache Push

We have a clear role for POST in FOREST: to push an updated object state to other objects that depend on it an object that is actually hosted by the source of the POST request and could otherwise be pulled or polled to meet those dependencies.

In this role POST is now idempotent, as it is effectively a “reverse GET”. We can push an object’s current state as much as we like in the same way that it could be repeatedly fetched idempotently by GET. To enable this idempotency and symmetry, the POSTed object must of course be self-descriptive enough to include its own URL.

Thus in FOREST a POST request is much the same as a GET response: it has a Content-Length, Content-Type and body. Since it carries the POSTed object’s URL, it can be pushed into the client-side cache of the target’s server for future GET requests. So to finish off this “POST request as GET response”, it could carry cache information, too.

The main cache parameters needed are Etag and max-age. These could go alongside the URL in the object itself, in a metadata section:

But, as mentioned above, these metadata parameters are candidates for pushing down into HTTP headers; into the POST request headers, unconventionally, with the URL as a “Content-Location” header:

Of course, this may cause issues, as unfortunately Cache-Control has a different meaning on a request: it could be seen as trying to constrain the POST response, rather than describe the POST request body or entity. If that is an issue for you, you could use the “Expires” header instead.

Note that, since such cache-pushing is an optimisation it is always possible to simply GET the object once it is been seen these cache parameters are optional, and could be ignored if discovered but not understood in the POST headers.

Optimising for Multiple Dependents on the Same Host

Now, what if there were two objects in one host that both depended on the same remote object? The framework driving those two objects could clearly save time and bandwidth by fetching the remote object dependency just once for both of them. It could put that object into the client-side cache, then show it to both dependents. It could then re-use the cached object immediately if a third object also looked at it.

Things get a little more complicated when the dependency pushes itself at those two objects. After doing this the first time, targeting each dependent, it would be wasteful to carry on POSTing twice per update into their common host.

Instead, it would be better if the object were pushed once, at one of the two objects, and, as in the GET case, put into the client-side cache, then distributed to the other dependents. That one POST request could be directed at the URL of one of the dependents at random, or at the dependent whose expected response is most interesting or needed most promptly.

But how does the host or framework of the remote object know that two of its dependents are on the same host, or share the same client-side cache? You cannot just assume two URLs with the same host:port actually point to the same back-end host; that could be a reverse proxy.

One solution is to return a unique cache identifier in the “Server” header of POST responses, to allow such correlation and bundling in future. Another solution is to add a new header in POST responses, perhaps “Cache-Notify”, which not only serves the same purpose of uniquely identifying that shared cache, but does that by offering a common URL that can be POSTed to, for all objects that return it. The object returned from a POST to this Cache-Notify URL would now be in the hands of that server perhaps the first dependent notified that reacted or updated. Almost equivalent to Cache-Notify would be a URL set up per incoming object effectively its URL in the cache. Then you would actually use PUT to set the state of that cache “resource”, and would not need the URL on the POSTed object itself.

For un-observing, in the first approach, a 403 could be returned if the object chosen lost interest, meaning the next POST should go to the next dependent sharing that Server header. In the second and third approaches, a 403 would mean there were no more dependents of the POSTed object currently left on this host, being served by that Cache-Notify or per-object URL.

All of this is just optimisation that would be implemented by the framework to keep it transparent to the object programmer, and none of it is required in FOREST.

Finally, the “Cache-Notify” header can be added to GET requests, to request a “subscription” to future changes in that target URL. This is a host-level agreement that does not involve any individual objects; the subscribed object would be unaware of the propagation of its updates. It is thus quite a different thing from the object-initiated form above. Indeed, it amounts to the generic form of network Publish-Subscribe. Fielding has rejected Publish-Subscribe on the Web (Fielding 2008), and this is an optional optimisation in FOREST that can be used to keep cluster caches fresh, etc.

Asymmetric “API”s

Sometimes you really are in an asymmetric situation and your clients are not able to host their own objects. Then the server side of this becomes more like an “API”. FOREST can support this asymmetric style of server-centric APIs.

The server carries on publishing its own Web of cacheable objects for the client to GET. The client can then continue to use Functional Observer to maintain the dependencies of its unpublished objects on those server-side objects.

But what about POSTs of those client objects, when there are no client URLs to include? How do server objects know that a POSTed object is the one they depend on and that they’ve seen before, without a URL to identify it?

The trick is simple: instead of the client sending the URL of its object in a POST, it just sends its unique id. Let us call that a “UID”, and say that it could be a UUID or a GUID. Here is what the Order POST would look like now:

The UID is prefixed with “uid-” to make its “type” immediately clear. Now, as we see here, with this approach we have to return the Ticket for this Order immediately. The rest of the interaction is much the same, only with more polling:

At payment time, you POST the Order, then the Payment:

Hence, there is a “pseudo hyperdata” coming from the client, since it cannot publish any objects. The server may cache those pushed objects keyed by their UID, and present a programming model that makes them look the same as other, published object resources.

On the subject of asymmetry, note that it is also theoretically possible to tunnel a 100% RESTful state-transfer protocol backwards through HTTP using various approaches.

Data Editing API

You may think you need an API to allow direct editing of a server’s resources. But in the majority of cases, it is better to think a level higher to think in terms of what you actually want to do in domain or application terms, and design interacting objects that declare their own higher-level meaning and intents. That is where the Functional Observer and FOREST approaches work best.

But what if you really do want a data editing service, offered through a traditional, asymmetric API? Using HTTP RESTfully using the traditional, four-method protocol style typified by AtomPub runs the designer into issues of ownership and partial ownership of data, responsibility for integrity, server-driven hypermedia, etc. For example, when suggesting an edit of a server resource using PUT, the client has to round-trip the entire representation it received, with all the server’s links, and all the content that it may not even understand. It drops its own edits somewhere in the middle and then lets the server sort it all out.

FOREST offers an alternative solution. But since FOREST pushes all the domain-level interactions up into the content in Functional Observer exchanges and only uses HTTP for pulling and pushing data, it is not obvious how to create, update and delete data, or how to manage optimistic locking. These are all traditionally performed in REST approaches through PUT, DELETE, PATCH, If-Match/Precondition Failed, etc., in the HTTP layer. What are the equivalents to these up in the content or domain layer, where FOREST has just state declarations  which are anyway incompatible with such imperative editing commands?

There is limited space in a single chapter to describe the full interaction here, but to summarise it: we should agree on a schema or syntax for describing “idempotent edit intentions” in objects that are then POSTed to a target object to be edited. These declarative intentions say things like: “I think you are in this state, and I’d like you to move into this state”, or “this is what state I want you to have, regardless of your current state”, or “please have this bit of data, that I believe is here currently, appear over there, instead”. All of these, being idempotent, can be POSTed repeatedly; those that specify current state are simply ignored if there is no match.

The great advantage of handling data editing up in the domain or content layer is that that layer has full insight into the nature of the data being edited, and can thus make intelligent decisions about incoming requests. For example, it may be possible to merge a “late” edit request instead of rejecting it as out of sync, or perhaps to partially apply it if within the integrity constraints and expectations of the domain. Otherwise, version synchronisation is still dealt with in the content in a similar way to the “not-as-ordered” flag in the fulfilment example above.

But then again, if you are thinking in such high-level domain terms, are you sure you want a low-level, direct, data-editing interaction, anyway? Just let the client express domain-level (rather than syntax-level) declarations and intentions directly!

User Objects

What if a user wanted to play in this object Web? Indeed, suppose that user, quite reasonably, wanted to use a browser to access it? How does FOREST’s object resource interaction model work then?

In MVC terms, the raw domain Model objects of FOREST will need some decoration, some transformation into a View. Serialising them into XHTML would allow them to be rendered reasonably well in their raw state then perhaps made more presentable by including some Javascript and CSS.

If serialised into JSON, they can be rendered to a View by a “FOREST browser” Web page, which would include some Javascript that knows how to navigate through and present the object Web, in from a starting point, perhaps detecting common object types and giving extra relevant interactivity for them.

On the POST side, or the Controller of MVC, the object in the application state that would be pushed back at the objects being rendered depends on those objects and their interaction specification.

The centre of any browser-hosted segment of object Web, however, is a first-class User object. This object can hold the user’s identity, vCard, chat status, etc., any of which could be POSTed to objects that are interested.

This whole area is far too broad to discuss in any depth in this single chapter, as it takes us into the rich subjects of Web Applications, Widgets, Ajax, Forms, HTML5, Mashups, etc. Suffice to say that FOREST is an excellent foundation for all that.

Programming Functional Observer

Functional Observer object resources are masters of their own evolution, never told what to do by other objects in the hyperdata graph. They determine their own state by looking around. These objects take responsibility themselves to pull and push state, guided by the domain logic for the class to which they belong.

All the pulling and pushing can happen completely automatically and transparently an object just has to focus on meeting its domain rules, and all observed objects visible through links, or links to links, are fetched when needed; plus, every time that object updates, its Etag is incremented and its state is pushed out to its local and some of its remote dependents.

This programming model brings the focus right down to what is important an object taking care of meeting its own domain constraints and state evolution given what it can see of itself and others through its links to them, and letting the framework take care of the state transfers in and out needed to support that. It allows very expressive declarative and reactive programming styles to be used. As mentioned above, this has affinity to the models available in Clojure and Erlang.

A great advantage of leaving the state transfer up to the framework is that it can handle not only the cacheing of remote objects, but also making local objects look much the same as remote ones, apart from the handling of timeouts and retries. This is most effective with an asynchronous and reactive programming model. Treating local objects like remote ones eases the further partitioning of an application. Clearly, it would make sense, in this light, to de-serialise remote object representations into their “solid” equivalent objects in an object cache.

This declarative programming model also leads quite naturally to the “eventual consistency” optimisation seen in large-scale systems. Instead of a workflow being kept in lock-step through an imperative model, we can relax and let the system “settle when it is ready”. A relative disposition of object states at any time is either fully resolved, or “in tension” needing further state evolution to reach the domain constraints being applied to each object involved.

Design Guidelines for FOREST

FOREST can be characterised as an informal list of design guidelines:

Functional Observer:
  • Implement domain logic as functional dependencies between objects’ state.

  • Ensure object structure conforms to common, shared standards.

  • An object’s next state depends on its current state plus the observed state of other linked objects.

  • Ensure objects are masters of their own evolution guided by other linked objects.

  • Discover these objects through links and links to links.

  • Alternatively discover objects via initial incoming push, leading to a new link.

  • Implement interactions using the Observer Pattern with observation via pull and push.

  • Guide the pulling and pushing of interdependent objects by domain logic over object interaction specs.

  • Push new state out to all observers and any objects that should be interested according to the domain logic.

  • Ensure an object cannot see the list of its observers.

  • Ensure objects only take what they need from their observed objects.

  • Use timeouts on pull and push that depend on domain context.

  • Consider using or writing a framework that automatically handles the pull and push of dependencies locally and remotely, automatically setting new Etags and propagating new state.

  • Drive appropriate objects partly or fully by external processes and interfaces.

Hyperdata:
  • Expose domain and application state via HTTP as your object resources with ids mapped to URLs.

  • Choose a common, base, data-oriented object serialisation or representation format from XML, XHTML1/5 or JSON.

  • Within that format, ensure your objects conform to standard or pre-existing schemas where possible.

  • Render links between objects as URLs in the serialisation, using links of the Media Type, if any, or maybe using XLink.

  • Thus link up objects within and across applications into a global object Web.

  • Separate responsibilities in your applications by partitioning your object space.

  • Treat local and cached remote objects the same, to ease ongoing partitioning.

State Transfer:
  • Use HTTP statelessly and only as an object representation state transfer protocol.

  • Use standard HTTP headers and return codes to drive state transfer only.

  • Use GET to pull remote objects on which local objects depend, following links.

  • Use POST to push updated object state idempotently to known dependent objects.

  • When POSTing, include the object’s URL in its serialisation or in the POST headers.

  • Use polling, max-age, retries and timeouts on both GET and POST as required.

  • Return any new state of the target on a POST response, whether 200 or 303.

  • Otherwise, return 204, 403, 405, etc., as required.

  • De-serialise remote objects and cache only their object forms.

  • Drive optimal cacheing by using HTTP headers correctly; return 304s, etc.

  • Use proxies and proxy-caches where beneficial.

Optional:
  • Consider using declarative, functional, reactive or other asynchronous programming styles over the Functional Observer framework.

  • Consider using eventual consistency where appropriate.

  • Consider exploiting cache information on POSTed objects held either in the object itself or the POST headers.

  • Consider, as an optimisation, consolidating cache notification using single POSTs to a Cache-Notify URL.

  • Consider, as an optimisation, asking for cache updates on GETs using Cache-Notify.

In short: design your application as a graph of objects that interact through the Functional Observer Pattern: write your application domain logic such that the state of those objects depends on their current state plus the state of those other objects they can “see” through links. Transfer that state by either pull or push. Integrate or distribute applications by publishing objects into a shared object Web via ids mapped to URLs and state mapped to a generic Media Type, with inter-object links. Then map object observation by pull and push to GET and POST respectively.

Benefits of the FOREST Approach

Here is a list of advantages of choosing the FOREST approach:
  • Functional Observer means focusing entirely on each object’s point of view with respect to surrounding objects.

  • Functional Observer enables easy declarative, functional, reactive, asynchronous programming styles.

  • RESTfulness easily attained by distribution of the Functional Observer object interaction pattern using HTTP.

  • Any “REST API” essentially drops out of the object interactions.

  • No threading, concurrency and locking issues objects manage their own state in a naturally parallel way.

  • Easy to separate concerns or partition across servers naturally parallel processing model without tight application boundaries.

  • Symmetric distribution across hosts acknowledges clients as first class servers and servers as clients.

  • Similar code whether a local or remote interaction just timeouts and retry logic may differ.

  • Good separation of concerns between HTTP state transfer and domain-level Functional Observer interactions.

  • Conflict resolution dealt with up in the domain or application by richer business logic, not at the HTTP level.

  • HTTP given simple state transfer role; observation maps to just the widely-used GET and POST.

  • Use of POST can be more efficient than polling; POST can update caches.

  • Gives “mashability” by exposing objects and linking them up across servers or applications.

  • Web-ready public state, in well-understood, data-oriented type such as JSON, XML or XHTML.

  • Requires stable, well-understood object types, structures and schemas, encouraging interoperability, re-use and mashability.

  • Simple linking model of unique object ids and inter-object links as URLs.

  • Creates hyperdata or an object Web efficient, semantic.

  • Eliminates URL design as URLs are opaque object ids; object Web is traversed through content semantics.

  • Object-based partitioning allows fine-grained cache control; only transfer data that is needed or that is changed.

  • Allows easy implementation of eventual consistency models for partitioning and scaling.

  • Allows independent evolution and loose coupling of each distributed application through the Stateless, Self-Descriptive Media Types and Hypermedia Constraints.

  • Delivers scalability through the Cache, Layered and Stateless Constraints.

Conclusion

FOREST describes a model of object interaction called a “Functional Observer Pattern”. FOREST’s Web of objects interact by setting their next state as a domain logic Function of their current state plus the states of other objects on which they depend, near them in the Web, Observed by pull or by push.

FOREST is distributed by a simple “Symmetric REST” architectural style using just GET and an idempotent POST to transfer object resource state in each direction between interlinked, interdependent objects from applications or servers being integrated or distributed. Clients host their own objects, or servers can become clients.

FOREST’s Web of object resources can be described as “hyperdata”. Its interpretation of REST’s Hypermedia Constraint can be stated as the more symmetric “Hyperdata as the Engine of Hyperdata”. This is the distributed form of the declarative object interdependence derived from the Functional Observer model.

FOREST can be seen as a symmetric interpretation of REST and Web Architecture for two-way, dynamic data scenarios, rather than the usually one-way, static documents of the Web. In general, when REST is re-interpreted for such SOA-like applications, the result can be called a “Resource-Oriented Architecture” or an “ROA”. FOREST can therefore be seen as a way of building a simple but powerful ROA for application integration or distribution.

Through being based on the Functional Observer Pattern, both in-process and across hosts, FOREST adds a number of advantages beyond the usual benefits it derives from naturally enabling a fully RESTful distributed architecture such as interoperability, scalability and evolvability. For example, FOREST is easy to program due to its declarative, functional, reactive and asynchronous nature. This enables business or domain logic to be encoded in terms of state dependencies and evolution. Further, it is easy to distribute, not just across application partitions, but across multicore, without the usual concerns around threads and locks.

FOREST can be implemented using traditional languages such as Java, but fits most easily and naturally over the programming models available in a language such as Clojure or Erlang, or over an asynchronous HTTP layer such as provided by Node.js.

References

  1. Armstrong J., Williams M., Wikstrom C. and Virding R. Concurrent Programming in Erlang, Prentice-Hall, Englewood Cliffs, NJ, USA. (1996). Online: http://www.erlang.org
  2. Berners-Lee, T., Fielding, R.T. et al. httpRange-14: What is the Range of the HTTP Dereference Function? (2010). Online: http://www.w3.org/2001/tag/issues.html#httpRange-14
  3. Berners-Lee, T., Fielding, R.T. et al. Uniform Resource Identifier (URI): Generic Syntax, (2005). Online: http://www.ietf.org/rfc/rfc3986.txt
  4. Cragg, D. Web Objects Ask, they Never Tell | The REST Dialogues, (2009). Online: http://duncan-cragg.org/blog/post/web-objects-ask-they-never-tell-rest-dialogues/
  5. Dahl, R. Node.js, (2009). Online: http://nodejs.org/jsconf.pdf
  6. Fielding, R.T. Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine, (2000). Online: http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
  7. Fielding, R.T. Post to the REST-Discuss Mailing List, (2006). Online: http://tech.groups.yahoo.com/group/rest-discuss/message/6613
  8. Fielding, R.T. Economies of Scale, (2008). Online: http://roy.gbiv.com/untangled/2008/economies-of-scale
  9. Fielding, R.T. et al. HTTP/1.1, Part 3: Message Payload and Content Negotiation, (2011). Online: http://tools.ietf.org/html/draft-ietf-httpbis-p3-payload-14
  10. Fielding, R.T. et al. Hypertext Transfer Protocol – HTTP/1.1, (1999). Online: http://www.ietf.org/rfc/rfc2616.txt
  11. Fielding, R.T. et al. HTTP/1.1, Part 2: Message Semantics, (2010). Online: http://tools.ietf.org/wg/httpbis/draft-ietf-httpbis-p2-semantics/
  12. Hickey, R. Clojure’s Approach to Identity and State, (2009). Online: http://qconlondon.com/dl/qcon-london-2009/slides/RichHickey_PersistentDataStructuresAndManagedReferences.pdf
  13. Mealling, M., Denenberg, R. Report from the Joint W3C/IETF URI Planning Interest Group: Uniform Resource Identifiers (URIs), URLs, and Uniform Resource Names (URNs): Clarifications and Recommendations, (2002). Online: http://www.ietf.org/rfc/rfc3305.txt

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.ThoughtWorks (UK) Ltd.LondonUK

Personalised recommendations