A Matter of Semantics: Structuring client-server communications with hypermedia messages

 Messages on the Web carry three levels of information: Structure Semantics, Protocol Semantics, and Application Semantics. No matter the implementation style, all three of these are needed for any successful communication between client and server. This threesome (S-P-A) forms the essentials of communication over distributed networks.

Most of the time, though, these levels are obscured or muddled at implementation time. For example, both Protocol Semantics (how we create valid network requests) and Application Semantics (domain names like users, customers, orders, etc) are often mixed together in conversation (“You POST new users to this URL”) and both of these are usually only defined in human-readable documentation and implemented in the source code of the client application itself. In other words, the protocol-level and application-level semantics are tightly coupled. An easy way to discover this is to see if you can take the same message format and implement your API using a protocol other than HTTP (e.g. WebSockets or FTP).  illustrated this “protocol-agnostic” design pattern back in 2010 (“A RESTful Hypermedia API in Three Easy Steps”).

But there is a way to keep these separate from each other and view each of these aspects in their own light. In doing so, you’ll strengthen the quality and value of your message design while increasing flexibility and choices.

Structure Semantics

Structure Semantics provides the set of rules regarding how to create a well-formed message. XML has rather simple structure semantics. JSON’s rules for well-formedness are a bit more vague but reachable since JSON.parse(…) turns out to be the ultimate arbiter of such things. Determining well-formedness of other, more complex media types (HTML, Atom, HAL, Collection+JSON) is tougher, but do-able even if external validators are not always available.

Building a successful web implementation that does not contain structure semantics is difficult—and that’s a good thing.

Protocol Semantics

Protocol Semantics is the set of rules regarding how to fashion valid requests and responses using one or more message protocols. HTTP, FTP, WebSockets, etc. are all message protocols. Most implementations for the Web use human-readable documentation to describe these semantics. For example, typical RPC-style implementations document URIs, associated request rules (HTTP methods and headers), and any possible body content.

Hypermedia-style implementations express the protocol semantics directly in the message (usually in the body, but sometimes in HTTP headers). The most common way this is done is to include rules for links and ‘forms’ that may appear in server responses (think HTML).

Registered media types don’t often include hypermedia controls. For example, CSV, XML, and JSON have no definitions for hypermedia expression within the message. However some media types such as HTML, Atom, HAL, Collection+JSON (and others), do include the predefined ability to express protocol-level information within the message sent from the server.

Application Semantics

Application Semantics govern the way in which application-specific information is shared between client and server. This includes names of domain data elements (users, customers, products, etc.) and specific actions within the domain (adding a user, calculating tax on purchases, etc.). Well-formed requests and responses are ones that contain valid expressions of these domain-specific concepts. There are very few registered media-type designs that contain this level of specificity and the ones that do contain it are usually aimed at an industry-wide domain (e.g. VoiceXML and Maze+XML).

There are some machine-readable examples of application semantics that can be used in conjunction with existing structurally semantic forms (XML, JSON, etc.). XSD, JSON-Schema, and other schema-based document types do a good job of expressing application-specific information. There are other examples such as WSDL for SOAP and its cousin WADL.

Most of the application-level semantics exist as stand-alone content with no standardized way to apply that information to a message. Some media type designs are better than others at supporting application-level semantics. HTML, for example, has several attributes (id, name, class, rel) with which you can apply application-level semantic information to a message.

Semantic Profiles

An alternative approach to application-level semantics is the use of a “Profile.” Tantek Celik has been championing this approach since 2003 with his XHTML Meta Data Profiles pattern for HTML. The “profile” link relation was recently registered with the IANA and there has been some renewed interest in this model. The POWDER spec uses the “describedby” link relation in a similar way, too.

The Profile pattern lets you outline the application semantics independent of the protocol and structure. You can collect up all the data points (userName, address, email, phone, etc.) and all the possible actions (addUser, editUser, approveUser) into a single place. The profile provides a clear, stand-alone description of the problem domain that can be kept up-to-date without mixing in protocol (HTTP, WS, etc.) or format (XML, Atom, HTML, etc.) details.

Even better, you can create a machine-readable version of this profile: one that can be referred to at run time by both client and server implementations. The Application-Level Semantic Profile (ALPS) format was designed for this purpose. This is an early attempt at a specification that supports profile descriptions that can be shared and applied to multiple media types (structures) and protocols. It’s too early to know if this kind of approach will become widely adopted, but I think it is worth pursuing.

Design Hallmarks

Designing a message format requires solving for all three levels (structure, protocol, and application). The simplest approach is to select an existing structural format (XML, JSON, etc.) and then use human-readable documentation to apply the other two (protocol and application). This requires added implementation coordination between client and server, but is quite common.

Selecting (or designing) media types that include protocol semantics and/or support applying application semantics directly within the message puts added responsibility on the message designer(s) and standardizes much more of the way client and servers interact. This can increase the likelihood that compatible implementations can be done at a distance (both in space and time) and that implementors (client and server developers) have an opportunity to safely evolve communications over time including adding new interactions to the domain-space without the need for prior coordination with the initial message designer.

These two aspects of the Web, support for distance and evolvability, are a hallmark of hypermedia-style designs.

Mongo DB vs Couchbase

NoSQL showdown: MongoDB vs. Couchbase

Which NoSQL database has richer querying, indexing, and ease of use?

Document databases may be the most popular NoSQL database variant of them all. Their great flexibility — schemas can be grown or changed with remarkable ease — makes them suitable for a wide range of applications, and their object nature fits in well with current programming practices. In turn, Couchbase Server and MongoDB have become two of the more popular representatives of open source document databases, though Couchbase Server is a recent arrival among the ranks of document databases.

In this context, the word “document” does not mean a word processing file or a PDF. Rather, a document is a data structure defined as a collection of named fields. JSON (JavaScript Object Notation) is currently the most widely used notation for defining documents within document-oriented databases. JSON’s advantage as an object notation is that, once you comprehend its syntax — and JSON is remarkably easy to grasp — then you have all you need to define what amounts to the schema of a document database. That’s because, in a document database, each document carries its own schema — unlike an RDBMS, in which every row in a given table must have the same columns.

More about NoSQL

There’s more than one way to break the RDBMS mold, so start with an overview of the CAP theorem and the NoSQL variants seeking to resolve it. Then find out what makes MongoDB a favorite among developers, and read an in-depth review of Oracle’s distributed, key-value datastore.

The latest versions of Couchbase Server and MongoDB are both newly arrived. In December 2012, Couchbase released Couchbase Server 2.0, a version that makes Couchbase Server a full-fledged document database. Prior to that release, users could store JSON data into Couchbase, but the database wrote JSON data as a blob. Couchbase was, effectively, a key/value database.

10gen released MongoDB 2.4 just this week. MongoDB has been a document database from the get-go. This latest release incorporates numerous performance and usability enhancements.

Both databases are designed to run on commodity hardware, as well as for horizontal scaling via sharding (in Couchbase, the rough equivalent to a shard is called a partition). Both databases employ JSON as the document definition notation, though in MongoDB, the notation is BSON (Binary JSON), a binary-encoded superset of JSON that defines useful data types not found in JSON. While both databases employ JavaScript as the primary data manipulation language, both provide APIs for all the most popular programming languages to allow applications direct access to database operations.

Key differences
Of course there are differences. First, MongoDB’s handling of documents is better developed. This becomes most obvious in the mongo shell, which serves the dual purpose of providing a management and development window into a MongoDB database. Database, collections, and documents are first-class entities in the shell. Collections are actually properties on database objects.

This is not to say that Couchbase is hobbled. You can easily manage your Couchbase cluster — adding, deleting, and fetching documents — from the Couchbase Management GUI, for which MongoDB has no counterpart. Indeed, if you prefer management via GUI consoles, score one for Couchbase Server. If, however, you favor life at the command line, you will be tipped in MongoDB’s direction.

The cloud-based MongoDB Monitoring Service (MMS), which gathers statistics, is not meant to be a full-blown database management interface. But MongoDB’s environment provides a near seamless connection between the data objects abstracted in the mongo shell and the database entities they model. This is particularly apparent when you discover that MongoDB allows you to create indexes on a specific document field using a single function call, whereas indexes in Couchbase must be created by more complex mapreduce operations.

In addition, while Couchbase documents are described via JSON, MongoDB documents are described in BSON; the latter notation includes a richer number of useful data types, such as 32-bit and 64-bit integer types, date types, and byte arrays. Both support geospatial data and queries, but this support in Couchbase is currently in an experimental phase and likely won’t stay there long. New in version 2.4, MongoDB’s full text search capability is also integrated with the database. A similar capability is available in Couchbase Server, but requires a plug-in for the elasticsearch tool.

Both Couchbase Server and MongoDB provide data safety via replication, both within a cluster (where live documents are protected from loss by the invisible creation of replica documents) and outside of a cluster (through cross data-center replication). Also, both provide access parallelism through sharding. However, where both Couchbase and MongoDB support hash sharding, MongoDB supports range sharding and “tag” sharding. This is a two-edged sword. On the one hand, it puts a great deal of flexibility at a database administrator’s fingertips. On the other hand, its misuse can result in an imbalanced cluster.

Mapreduce is a key tool used in both Couchbase and MongoDB, but for different purposes. In MongoDB, mapreduce serves as the means of general data processing, information aggregating, and analytics. In Couchbase, it is the means of creating indexes for the purpose of querying data in the database. (We suspect that this, like the poorer document handling, is an effect of Couchbase’s only having recently morphed into a document database.) As a result, it’s easier to create indexes and perform ad hoc queries in MongoDB.

Couchbase’s full incorporation of Memcached has no counterpart in MongoDB, and Memcached is a powerful adjunct as general object caching system for high-throughput, data-intensive Internet and intranet applications. If your application needs a Memcache server with your database, then look no further than Couchbase.

In general, the two systems are neck-and-neck in terms of features provided, though the ways those features are implemented may differ. Further, the advantages that one might hold over the other will certainly come and go as development proceeds. Both provide database drivers and client frameworks in all the popular programming languages, both are open source, both are easily installed, and both enjoy plenty of online documentation and active community support. As is typical for such well-matched systems, the best advice anyone could give for determining one over the other will be that you install them both and try them out.

Couchbase Server vs. MongoDB at a glance
Document handling Couchbase’s document database characteristics were added to its existing key/value storage architecture with the version 2.0 release. MongoDB was designed as a document database from the get-go. MongoDB’s handling of documents is better developed.
Indexing With Couchbase, you use mapreduce to create additional indexes in the database; these accelerate queries. With MongoDB, you can specify that individual document fields be indexed. Mapreduce in MongoDB is used for general processing.
Memcached Couchbase includes a Memcached component that can operate independently (if you wish) from the document storage components. MongoDB has no counterpart.
Sharding Couchbase supports only hashed sharding. MongoDB supports hashed sharding and range sharding.
Geospatial MongoDB has geospatial capabilities. Couchbase does too, but they were added in version 2.0 and are considered experimental.

Couchbase Server

Couchbase promotes Couchbase Server as a solution for real-time access, not data warehousing. Nor is Couchbase Server suitable for batch-oriented analytic processing — it is designed to be an operational data store.

Though Couchbase Server is based on Apache CouchDB, it is more than CouchDB with incremental modifications. For starters, Couchbase is an amalgam of CouchDB and Memcached, the distributed, in-memory, key/value storage system. In fact, Couchbase can be used as a direct replacement for Memcached. The system provides a separate port that unmodified, legacy Memcached clients can use, as well as “smart SDK” and proxy tools that improve its performance as a Memcached server.

For example, you can use a “thick client” deployment model, which will place the continuously updated knowledge of Memcached node topology on the client tier. This speeds response, as any request for a particular Memcached object will be sent from the client directly to the caching node for that object. This thick-client approach also plays an important role in the Couchbase system’s resilience to node crashes (described later).

Couchbase includes its own object-level caching system based on Memcached, though with enhancements. For example, Couchbase tracks working sets (the documents most frequently accessed on a given node) in its object cache using NRU (not recently used) algorithms. All I/O operations act on this in-memory cache. Updates to documents in the cache are eventually persisted to disk. In addition, for updates, locking is employed at the document level — not at the node, database, or partition level (which would hobble throughput with numerous I/O waits), nor at the field level (which would snarl the system with memory and CPU cycles required to track the locks).

Couchbase accelerates access by using “append only” persistence. This is used not only with the data, but with indexes as well. Updated information is never overwritten; instead, it is appended to the end of whatever data structure is being modified. Further, deleted space is reclaimed by compaction, an operation that can be scheduled to take place during times of low activity. Append-only storage speeds updates and allows read operations to occur while writes are taking place.

Couchbase scaling and replication
To facilitate horizontal scaling, Couchbase uses hash sharding, which ensures that data is distributed uniformly across all nodes. The system defines 1,024 partitions (a fixed number), and once a document’s key is hashed into a specific partition, that’s where the document lives. In Couchbase Server, the key used for sharding is the document ID, a unique identifier automatically generated and attached to each document. Each partition is assigned to a specific node in the cluster. If nodes are added or removed, the system rebalances itself by migrating partitions from one node to another.

There is no single point of failure in a Couchbase system. All partition servers in a Couchbase cluster are equal, with each responsible for only that portion of the data assigned to it. Each server in a cluster runs two primary processes: a data manager and a cluster manager. The data manager handles the actual data in the partition, while the cluster manager deals primarily with intranode operations.

System resilience is enhanced by document replication. The cluster manager process coordinates the communication of replication data with remote nodes, and the data manager process shepherds whatever replica data the cluster has assigned to the local node. Naturally, replica partitions are distributed throughout the cluster so that the replica copy of a partition is never on the same physical server as the active partition.

Like the documents themselves, replicas exist on a bucket basis — a bucket being the primary unit of containment in Couchbase. Documents are placed into buckets, and documents in one bucket are isolated from documents in other buckets from the perspective of indexing and querying operations. When you create a new bucket, you are asked to specify the number of replicas (up to three) to create for that bucket. If a server crashes, the system will detect the crash, locate the replicas of the documents that lived on the crashed system, and promote those replicas to active status. The system maintains a cluster map, which defines the topology of the cluster, and this is updated in response to the crash.

Note that this scheme relies on thick clients — embodied in the API libraries that applications use to communicate with Couchbase — that are in constant communication with server nodes. These thick clients will fetch the updated cluster map, then reroute requests in response to the changed topology. In addition, the thick clients participate in load-balancing requests to the database. The work done to provide load balancing is actually distributed among the smart clients.

Changes in topology are coordinated by an orchestrator, which is a server node elected to be the single arbiter of cluster configuration changes. All topology changes are sent to all nodes in the cluster; even if the orchestrator node goes down, a new node can be elected to that position and system operation can continue uninterrupted.

Couchbase supports cross-data-center replication (XDCR), which provides live replication of database contents of one Couchbase cluster to a geographically remote cluster. Note that XDCR operates simultaneously with intracluster replication (the copying of live documents to their inactive replica counterparts on other cluster members), and all systems in an XDCR arrangement invisibly synchronize with one another. However, Couchbase does not provide automatic fail-over for XDCR arrangements, relying instead on techniques such as using a load-balancing mechanism to reroute traffic at the network layer, in which case the XDCR group will have been set up in a master-master configuration.

Couchbase indexing and queries
Queries on Couchbase Server are performed via “views,” Couchbase terminology for indexes. Put another way, when you create an index, you’re provided with a view that serves as your mechanism for querying Couchbase data. Views are new to Couchbase 2.0, as is the incremental mapreduce engine that powers the actual creation of views. Note that queries really didn’t exist prior to Couchbase Server 2.0. Until this latest release, the database was a key/value storage system that simply did not understand the concept of a multifield document.

To define a view, you build a specific kind of document called a design document. The design document holds the JavaScript code that implements the mapreduce operations that create the view’s index. Design documents are bound to specific buckets, which means that queries cannot execute across multiple buckets. Couchbase’s “eventual consistency” plays a role in views as well. If you add a new document to a bucket or update an existing document, the change may not be immediately visible.

The map function in a design document’s mapreduce specification filters and extracts information from the documents against which it executes. The result is a set of key/value pairs that comprise the query-accelerating index. The reduce function is optional. It is typically used to aggregate or sum the data manipulated by the map operation. Code in the reduce function can be used to implement operations that correspond roughly to SQL’s ORDER BY, SORT, and aggregation features.

Couchbase Server supplies built-in reduce functions: _count, _stats, and _sum. These built-in functions are optimized beyond what would be possible if written from scratch. For example, the _count function (which counts the number of rows returned by the map function) doesn’t have to recount all the documents when called. If an item is added to or removed from the associated index, the count is incremented or decremented appropriately, so the _count function need merely retrieve the maintained value.

Query parameters offer further filtering of an index. For example, you can use query parameters to define a query that returns a single entry or a specified range of entries from within an index. In addition, in Couchbase 2.0, document metadata is available. The usefulness of this becomes apparent when building mapreduce functions, as the map function can employ metadata to filter documents based on parameters such as expiration date and revision number.

Couchbase indexes are updated incrementally. That is, when an index is updated, it’s not reconstructed wholesale. Updates only involve those documents that have been changed or added or removed since the last time the index was updated. You can configure an index to be updated when specific circumstances occur. For example, you might configure an index to be updated whenever a query is issued against it. That, however, might be computationally expensive, so an alternative is to configure the index to be updated only after a specified number of documents within the view have been modified. Still another alternative is to have the view updated based on a time interval.

Whatever configuration you choose, it’s important to realize that a design document can hold multiple views and the configuration applies to all views in the document. If you update one index, all indexes in the document will be updated.

Finally, Couchbase distinguishes between development and production views, and the two are kept in separate namespaces. Development views can be modified; production views cannot. The reason for the distinction arises from the fact that, if you modify a view, all views in that design document will be invalidated, and all indexes defined by mapreduce functions in the design document will be rebuilt. Therefore, development views enable you to test your view code in a kind of sandbox before deploying it into a production view.

NoSQL deep dive: MongoDB vs. Couchbase

You can manage Couchbase Server via the Web-based management console. The view of active servers, shown above, is open to a single member of the cluster. Memory cache and disk storage usage information is readily available.

Managing Couchbase
For gathering statistics and managing a Couchbase Server cluster, the Couchbase Web Console — available via any modern browser on port 8091 of a cluster member — is the place to go. It provides a multitab view into cluster mechanics. The tabs include:

  • Cluster overview, which has general RAM and disk usage information (aggregated for the whole cluster). Also, operations per second bucket usage. The information is presented in a smoothly scrolling line graph.
  • Server nodes, which provides information similar to the above, but for individual members of the cluster. You can also see CPU usage and swap space usage. On this tab, you can add a new node to the cluster: Click the Add Server button and you’re prompted for IP address and credentials.
  • Data buckets, which shows all the buckets on the cluster. You can see which nodes participate in the storage of a given bucket, how many items are in each bucket, RAM and disk usage attributed to a bucket, and so on.

The Couchbase Web Console provides much more information than can be covered here. An in-depth presentation of its capabilities can be found in Couchbase Server’s online documentation.

For administrators who would rather perform their management duties on the metal, Couchbase provides a healthy set of command-line tools. General management functions are found in the couchbase-cli tool, which lists all the servers in a cluster, retrieves information for a single node, initiates rebalancing, manages buckets, and more. The cbstats command-line tool displays cluster statistics, and it can be made to fetch the statistics for a single node (the variety of statistical information retrieved is too diverse to list here). The cbepctl command lets you modify a cluster’s RAM and disk management characteristics. For example, you can control the “checkpoint” settings, which govern how frequently replication occurs.

Other command-line tools include data backup and restore, a tool to retrieve data from a node that has (for whatever reason) stopped running, and even a tool for generating an I/O workload on a cluster member to test its performance.

Couchbase Server is available in both Enterprise and Community editions. The Enterprise edition undergoes more thorough testing than the Community edition, and it receives the latest bug fixes. Also, hot fixes are available, as is 24/7 support (with the purchase of an annual subscription). Nevertheless, the Enterprise edition is free for testing and development on any number of nodes or for production use on up to two nodes. The Community edition, as you might guess, is free for any number of production nodes.

Pros and cons: Couchbase Server 2.0
  • Provides legacy Memcached capabilities
  • Supports spatial data and views
  • Now a true document database
  • Indexing mechanisms not well developed
  • JSON support is relatively immature
  • Does not support range sharding
Review: Visual Studio 2012 shines on Windows 8


MongoDB is about three years old, first released in late 2009. The goal behind MongoDB was to create a NoSQL database that offered high performance and did not cast out the good aspects of working with RDBMSes. For instance, the way that queries are designed and optimized in MongoDB is similar to how that would be done in an RDBMS. MongoDB’s designers also wanted to make the database easier for application developers to work with — for example, by allowing developers to change the data model quickly. MongoDB, whose name is short for “humongous,” stores documents in BSON (Binary JSON), an extension of JSON that allows for the use of data types such as integers, dates, and byte arrays.

Two primary processes are at work in a MongoDB system, mongod and mongos. The mongod process is the real workhorse. In a sharded MongoDB cluster, mongod can be found playing one of two roles: config server or shard server. The config server tracks the cluster’s metadata. (In a sharded MongoDB cluster, there must be at least three config servers for redundancy’s sake.) Each config server knows which server in the cluster is responsible for a given document or, more precisely, where a given contiguous range of shard keys (called a chunk) belongs in the cluster.

Other mongod processes in the cluster run as shard servers, and these handle the actual reading and writing of the data. For fail-over purposes, two instances of a mongod process on a given cluster member run as shard servers. One process is primary, and the other is secondary. All write requests go to the primary, while read requests can go to either primary or secondary.

Secondaries are updated asynchronously from the primary so that they can take over in the event of a primary’s crash. This, however, means that some read requests (sent to secondaries) may not be consistent with write requests (sent to primaries). This is an instance of MongoDB’s “eventual consistency.” Over time, all secondaries will become consistent with write operations on the primary. Note that you can guarantee consistent read/write behavior by configuring a MongoDB system such that all I/O — reads and writes — go to the primary instances. In such an arrangement, secondaries act as standby servers, coming online only when the primary fails.

The mongos process, which runs at a conceptually higher level than the mongod processes, is best thought of as a kind of routing service. Database requests from clients arrive first at a mongos process, which determines which shard(s) in a sharded cluster can service each request. The mongos process dispatches I/O requests to the appropriate mongod processes, gathers the results, and returns them to the client. Note that in a nonsharded cluster, clients talk directly to a mongod process.

MongoDB scaling and replication
MongoDB doesn’t have an explicit memory caching layer. Instead, all MongoDB operations are performed through memory-mapped files. Consequently, MongoDB hands off the chore of juggling memory caching versus persistence-to-disk to the operating system. You can tweak various flush-to-disk settings for optimal performance, however. For example, MongoDB maintains a journal of database operations (for recovery purposes) that is flushed to the disk every 100ms. Not only is this interval configurable, but you can configure the system so that write operations return only after the journal has been written to disk.

Documents are placed in named containers called collections, which are roughly equivalent to Couchbase’s buckets. A collection serves as a means of partitioning related documents into separate groups. The effects of many multidocument operations in a MondoDB database are restricted to the collection in which those operations are performed. MongoDB supports sharding at the collection level, which means — should requirements dictate — you could construct a database with unsharded and sharded collections. Of course, only a sharded collection is protected against a single point of failure.

A document’s membership in a particular shard is determined by a shard key, which is derived from one or more fields in each document. The exact fields can be specified by the database administrator. In addition, MongoDB provides autosharding, which means that, once you’ve configured sharding, MongoDB will automatically manage the storage of documents in the appropriate physical location. This includes rebalancing shards as the number of documents grows or the number of mongod instances changes.

As of the 2.4 release, MongoDB supports both hash-based sharding and range-based sharding. As you might guess, hash-based sharding hashes the shard key, which creates a relatively even distribution of documents across the cluster. With range-based sharding (the sole sharding type prior to 2.4), a given member of a MongoDB sharded cluster will store all the documents within a given subrange of the shard key’s overall domain. More precisely, MongoDB defines a logical container, called a chunk, which is a subset of documents whose shard keys fall within a specific range. The mongos process then dictates which mongod process will manage a given chunk.

Typically, you permit the load balancer to determine which cluster member manages a given shard range. However, with version 2.4, you can associate tags with shard ranges (a tag being nothing more than an identifying string). Once that’s done, you can specify which member of a cluster will manage any shard ranges associated with a tag. In a sense, this lets you override some of the load balancer’s decision making and steer identifiable subsets of the database to specific servers. For example, you could put the data most frequently accessed from California on the cluster member in California, the data most frequently accessed from Texas on the cluster member in Texas, and so on.

MongoDB’s locking is on the database level, whereas it was global prior to version 2.2. The system implements shared-read, exclusive-write locking (many concurrent readers, but only one writer) with priority given to waiting writers over waiting readers. MongoDB avoids contentions via yield operations within locks. Predictive coding was added to the 2.2 release; if a process requests a document that is not in memory, it yields its lock so that other processes — whose documents are in memory — can be serviced. Long-running operations will also periodically yield locks.

You’ll find no clear notion of transactions in MongoDB. Certainly, you cannot perform pure ACID transactions on a MongoDB installation. Database changes can be made durable if you enable journaling, in which case write operations are blocked until the journal entry is persisted to disk (as described earlier). And MongoDB defines the $atomic isolation operator, which imposes what amounts to an exclusive-write lock on the document involved. However, $atomic is applied at the document level only. You cannot guard multiple updates across documents or collections.

MongoDB indexing and queries
MongoDB makes it easy to create secondary indexes for all document fields. A primary index always exists on the document ID. As with Couchbase Server, this is automatically generated for each document. However, with MongoDB, you can specify a separate field as being the document’s unique identifier. For example, a database of bank accounts might use the bank’s generated account number as the document ID field. Indexes exist at the collection level, and they can be compound — that is, created on multiple fields. MongoDB can also handle multikey indexes. If you index a field that includes an array, MongoDB will index each value in that array. Finally, MongoDB supports geospatial indexes.

MongoDB’s querying capabilities are well developed. If you’re coming to MongoDB from the RDBMS world, the online documentation shows how SQL queries might be mapped to MongoDB operations. For example, in most cases, the equivalent of SQL’s SELECT can be performed by a find() function. The find() function takes two arguments: a query document and a projection document. The query document specifies filter operations on specific document fields that are fetched. You could use it to request that only documents with a quantity field whose contents are greater than, say, 100 be returned. Therefore, the query document corresponds to the WHERE clause in an SQL statement. The projection document identifies which fields are to be returned in the results, which allows you to request that, say, only the name and address fields of matching documents be returned from the query. The sort() function, which can be executed on the results of find(), corresponds to SQL’s ORDER BY statement.

You can locate documents with the command db.<collection>.find(), possibly the simplest query you can perform. The find() command will return the first 20 members of the result, but it also provides a cursor, which allows you to iterate through all the documents in the collection. If you’d like to navigate the results more directly, you can reference the elements of the cursor as though it were an array.

More complex queries are possible thanks to MongoDB’s set of prefix operators, which can describe comparisons as well as boolean connections. MongoDB also provides the $regex operator in case you want to apply regular expressions to document fields in the result set. These prefix operators can be used in the update() command to construct the MongoDB equivalent of SQL’s UPDATE ... WHERE statement.

In the 2.2 release, MongoDB added the aggregation framework, which allows for calculating aggregated values without having to resort to mapreduce (which can be overkill if all you want to do is calculate a field’s total or average). The aggregation framework provides functionality similar to SQL’s SUM and AVG functions. It can also calculate computed fields and mimic the GROUP BY operator. Note that the aggregation framework is declarative — it does not employ JavaScript. You define a chain of operations, much in the same way you might perform Unix/Linux shell programming, and these operations are performed on the target documents in stream fashion.

One of the more significant new features in MongoDB’s 2.4 release is the arrival of text search. In the past, developers accomplished this by integrating Apache Lucene with MongoDB, which piled on considerable complexity. Adding Lucene in a clustered system with replication and fault tolerance is not an easy thing to do. MongoDB users now get text search for free. The new text search feature is not meant to match Lucene, but to provide basic capabilities such as more efficient Boolean queries (“dog and cat but not bird”), stemming (search for “reading” and you’ll also get “read”), and the automatic culling of stop words (for example, “and”, “the”, “of”) from the index.

You can define a text index on multiple string fields, but there can be only a single text index per collection, and indexes do not store word proximity information (that is, how close words are to one another, which can affect how matches are weighted). In addition, the text index is fully consistent: when you update data, the index is also updated.

Ease-of-use features have been added to version 2.4 as well. For example, you can now define a “capped array” as a data element, which works sort of like an ordered circular buffer. If, for example, you’re keeping track of the top 10 entries in a blog, using a capped array will allow you to add new entries, and (based on the specified ordering) previous entries will be removed to cap the array at 10 or whatever number you specify.

MongoDB 2.4 also has improved geospatial capabilities. For example, you can now perform polygon operations, which would allow you to determine if two regions overlap. The spherical model used in 2.4 is improved too; it now takes into account the fact that the earth is not perfectly spherical, so distance calculations are more accurate.

MongoDB mapreduce
In Couchbase Server, the mapreduce operation’s primary job is to provide a structured query and information aggregation capability on the documents in the database. In MongoDB, mapreduce can be used not only for querying and aggregating results, but as a general-purpose data processing tool. Just as a mapreduce operation executes within a given bucket in Couchbase Server, mapreduce executes within a given collection in a MongoDB database. As in Couchbase Server, mapreduce functions in MongoDB are written in JavaScript.

You can filter the documents passed into the map function via comparison operators, or you can limit the number of documents to a specific number. This allows you to create what amounts to an incremental mapreduce operation. Initially, you run mapreduce over the entire collection. For subsequent executions, you add a query function that includes only newly added documents. From there, set the output of mapreduce to be a separate collection, and configure your code so that the new results are merged into the existing results.

Further speed/size trade-offs are possible by choosing whether the intermediate results (the output of the map function, sent to the reduce function) are converted to BSON objects or remain JavaScript objects. If you choose BSON, the BSON objects are placed in temporary, on-disk storage, so the number of items you can process is limited only by available disk space. However, if you choose JavaScript objects, then the system can handle only about 500,000 distinct keys emitted by the map function. But as there is no writing to disk, the processing is faster.

You have to be careful with long-running mapreduce operations, because their execution involves lengthy locks. As mentioned earlier, the system has built-in facilities to mitigate this. For example, the read lock on the input collection is yielded every 100 documents. The MongoDB documentation describes the various locks that must be considered — as well as mechanisms to relieve the possible problems.

Managing MongoDB
Management access with the MongoDB database goes through the interactive mongo shell. Very much a command-line interface that lets you enter arbitrary JavaScript, it is nonetheless surprisingly facile. The MongoDB related commands are uncomplicated, but at the price of being dangerous if you’re careless. For example, to select a database, you enter use <databasename>. But that command doesn’t check for the presence of the specific database; if you mistype it and proceed to enter documents into that database, you might not know what’s going on until you’ve put a whole lot of documents into the wrong place. The same goes for collections within databases.

Other useful command-line utilities are mongostat, which returns information concerning the number of operations — inserts, updates, deletes, and so on — within a specific time period. The mongotop utility likewise returns statistical information on a MongoDB instance, this time focusing on a specific collection. You can see the amount of time spent reading or writing in the collection, for instance.

In addition, 10gen provides the free cloud-based MongoDB Monitoring Service (MMS) which provides a monitoring dashboard for MongoDB installations. Built on the SaaS model, MMS requires you to run a small agent on your MongoDB cluster that communicates with the management system.

NoSQL deep dive: MongoDB vs. Couchbase

10gen’s MongoDB Monitoring Service shows statistics — in this case, for a replica set — but management of the database is done from the command line.

In addition to the new text search and geospatial capabilities discussed above, MongoDB 2.4 comes with performance and security improvements. The performance enhancements include the working set analyzer. The idea is that you want to configure your system so that the working set — that subset of a databas accessed most frequently — fits entirely in memory. But it was not easy to figure out your working set or how much memory you need. The working set analyzer, which operates like a helper function, provides diagnostic output to aid you in discovering the characteristics of your working set and tuning your system accordingly. In addition, the JavaScript engine has been replaced by Google’s open source V8 engine. In the past, the JavaScript engine was single-threaded. V8 permits concurrent mapreduce jobs, as well as general speed improvements.

Finally, the Enterprise edition welcomes Kerberos-based authentication. In all editions, MongoDB now supports role-based privileges, which gives you finer-grained control over users’ access and operations on databases and collections.

10gen’s release of MongoDB 2.4 is accompanied by new subscription levels: Community, Basic, Standard, and Enterprise. The Community subscription level is free, but it’s also free of any support. The other subscription levels provide varying support response times and hours of availability. In addition, the Enterprise subscription level comes with the Enterprise version of MongoDB, which has more security features and SNMP support. It has also undergone more rigorous testing.

Pros and cons: MongoDB 2.4
  • New release incorporates text search
  • New release adds improved JavaScript engine
  • Free MongoDB training courseware available from 10gen
  • Text index doesn’t store proximity information
  • No GUI-based management console
  • Kerberos authentication available in Enterprise edition only

Mongo or Couch?
As usual, which product is the best choice depends heavily on the target application. Both are highly regarded NoSQL databases with outstanding pedigrees. On the one hand, MongoDB has spent much more of its lifetime as a document database, and its support for document-level querying and indexing is richer than that in Couchbase. On the other hand, Couchbase can serve equally well as a document database, a Memcached replacement, or both.

Happily, exploring either Couchbase or MongoDB is remarkably simple. A single-node system for either database server is easily installed. And if you want to experiment with a sharded system (and have enough memory and processor horsepower), you can easily set up a gang of virtual machines on a single system, and lash them together via a virtual network switch. The documentation for both systems is voluminous and well maintained. 10Gen even provides free online MongoDB classes for developers, complete with video lectures, quizzes, and homework.

Open your data to the world

Open your data to the world

Not having an API is becoming like not having a website

When Neil Fantom, a manager at the World Bank, sat down with the organization’s technology team in 2010 to talk about opening up the bank’s data to the world at large, he encountered a bit of unfamiliar terminology. “At that time I didn’t even know what ‘API’ meant,” says Fantom.

As head of the bank’s Open Data Initiative, announced in April 2010, Fantom was in charge of taking the group’s vast trove of information, which previously had been available only by subscription, and making it available to anyone who wanted it. The method of doing that, he would learn, would be an application programming interface.

 The API would place thousands of economic indicators, including rainfall amounts, education levels and birth rates — some metrics going back 50 years — at the disposal of developers to mix and match and present in any way that made sense to them. The hope was that this would advance the bank’s mission of fighting poverty on a global scale by calling on the creativity of others. “There are many people outside the bank who can do things with the data set we never thought about,” says Fantom.

One developer, for instance, created an app that married the bank’s rainfall data to Google Maps to estimate how much rainwater could be collected on rooftops and subsequently used to water crops in different parts of the world. Another app provides facts about energy consumption and shows individuals what they can do to fight climate change.

Fantom and the World Bank aren’t alone in this trajectory. A decade ago, open APIs were a novelty, but in the last few years they’ve been adopted at an accelerating rate. ProgrammableWeb, a website that tracks public APIs, listed more than 8,800 in early April. According to the site’s data, it took from 2000 to 2008 for the number of APIs to reach 1,000, and then another 18 months for that to double. The jump from 7,000 to 8,000 took just three months.

The APIs cover a wide range of categories, including business, shopping, messaging, mapping, telephone, social, financial and government, according to the ProgrammableWeb website. They’re becoming as necessary to an organization as a website. “In business today, an open API is more or less table stakes. It’s something you have to have,” says Stephen O’Grady, an analyst at RedMonk, an analysis firm that focuses on developers. “Increasingly, your traction is going to be driven by how open and how programmatically manipulable your product is.”

When Best Buy first launched its API, BBYOpen, in 2009, it gave developers access only to the chain’s products catalog, with descriptions and prices for all the items it had on sale, in the hopes that doing so would bring in more customers. That was part of a deliberate strategy to start slowly, says Steve Bendt, director of emerging platforms at Best Buy. “We had to prove these things over time,” he says. “We started to prove out that this is a very vibrant and viable area to pursue.”

But external developers wanted more, so the company added the ability to access reviews and ratings for products, find a nearby store and check whether a product is available there, and purchase the item through the website or mobile app in question, perhaps with a single click if the user has linked a credit card to the app.

It’s been a hit. The mobile apps ShopSavvy, RedLaser and Milo all use BBYOpen as part of their apps. The makers of the app get a commission on sales through Best Buy’s affiliate program. Shoppers can search for an item, or scan a bar code, and get information on pricing from various sellers.

Of course, that might mean a customer using the app might wind up buying from a competitor instead, but Bendt says that since websites and mobile apps have changed how people shop, what’s important for Best Buy is to be in the mix. “If we’re not in the consideration set, that’s a missed opportunity.” And the fact that the API makes it possible for customers to find out if a product is available for pickup at a nearby store once they’ve purchased it helps provide a competitive edge over online-only retailers, he says. “Now you can search for, buy and pick up within a matter or 20 to 40 minutes.”

Legacy data issues
The idea of an in-store pickup option actually came from external developers, Bendt says, and it took the chain some effort to adapt its legacy system to make inventory data available through the API; the data needed to be reformatted to be compatible. “The systems were built at a time before web services and APIs were in active use,” he explains. “It wasn’t built in a way to expose it externally to the developer.”

The specifics of how they did that varied greatly depending on the data source, but generally the team would try to expose some “snapshot” of the data, updated as frequently as possible. If the data proved useful, they found ways to make it available in closer to real time.

Getting existing systems to work with the new API was also a challenge at the World Bank, says Malarvizhi Veerappan, open data systems lead. Her group originally struggled with latency issues because their 8,000 different economic indicators were not all directly linked to one another. It was important, she says, to create a structure that could incorporate all that historical data and grow as new information accumulated.

“We didn’t want the API to be a separate application. We wanted it to be part of everything else we did with the data,” she says. “We needed to connect it back to our data system. It did require our improving our internal data system.”

As the API grew, the team added performance monitoring and instituted policies to ensure good traffic flow. The organization also increased server capacity and added server redundancy to assure availability of the API.

When financial information provider Bloomberg LP launched its Open Data Initiative in February 2012, the new open API — BLPAPI — was actually version 3 of the software development kit the company had already been using internally, says Shawn Edwards, Bloomberg’s chief technology officer. In the old days, Bloomberg customers were given a dedicated terminal that connected them to the company’s mainframe, which delivered market data, news and analysis.

(The name “Open Data Initiative” for both the World Bank and Bloomberg projects is just a coincidence; neither has any formal relationship with the Open Data Initiative that is about making use of publicly available data from various government sources.)

Bloomberg’s project has since evolved into a software package that customers install on their own systems. Even before making it open, the company used the API to develop specific applications that allow customers to manipulate Bloomberg data on their own desktops.

With the launch of its open API, the company is now allowing customers to create their own apps, such as watch lists for selected securities or their own trading systems. They also allow outside developers to create apps that draw on other data sources as well as Bloomberg’s. “We’re not giving away market data. What this allows people to do is integrate with other services,” Edwards says. “The API is a piece of software that connects to the Bloomberg cloud.”

It just makes sense to let others do the app development, he explains. “We’re not in the business of selling software,” he says. “We’re going to win their business by providing the best services and the best data.”

When Bloomberg put out the open API, it decided to remove some of the old features that the previous versions supported. There was discussion as to whether the API should be backward compatible. “We said no,” Edwards says. That meant some customers wound up with deprecated functions, but Edwards says it makes the API less cluttered with out-of-date functions.

Like most open APIs, the BLPAPI supports a variety of languages, so developers can choose the best one for their app. Someone running an overnight batch process might choose Perl, or the recently released Python version. An electronic trading system would probably run on C or C++. Quantitative analysts, or quants, generally use the data in Matlab. The API also supports Java, .Net, and C#, and Edwards says some developers are using an R wrapper as well.

One key to making an API successful lies in making it easy to use. Back in 2000, RedMonk’s O’Grady says, APIs often used web services protocols, but they proved too complex. Now about three quarters of all APIs are REST-based, according to ProgrammableWeb, with SOAP a distant second. “Because developers overwhelmingly preferred this, it’s now the dominant protocol for API systems,” O’Grady says.

The importance of clarity
Another important requirement is having extensive, clear documentation, and tools to help developers do their jobs. Bloomberg’s initial documentation was aimed more at the financial experts who are its customers, and had to be reworked to tell developers what they needed to know.

The BLPAPI also tried to make work easier for developers by providing a replay tool that allows them to perform trial runs of their apps, but that was not available when it first launched. Best Buy’s BBYOpen also gives developers a set of tools, including a test console to run apps and an automatic widget generator. The World Bank offers a query builder that lets developers select options.

Tools and ideas don’t all flow outward from the organizations; external developers often provide information and frameworks to help each other out. BBYOpen, for instance, offers libraries created by developers in Java, .NET, PHP and other languages. At the World Bank, there’s a discussion forum where developers can ask questions, and others jump in with solutions.

“They don’t wait for us to respond to questions in the forum,” says Veerappan, who is working on giving the forum more features and converting it into a knowledge base. “It’s kind of interesting to see the knowledge that other developers have gained in the API,” she says.

Successful APIs tend to have MIT-style open-source software licenses; the World Bank, for example, uses an Open Source Attribution License. O’Grady says one key to success is being very clear about the terms of service, and not having an overly restrictive license that discourages use.

He says Stack Overflow, a collaboratively edited question-and-answer site for programmers, has a very nice API, for instance, but that the terms of using it are difficult to navigate. Twitter irritated some developers, he adds, by being too insistent about issues such as how the time stamp was formatted, or insisting that the word “tweet” must be capitalized. While developers are unlikely to shun Twitter for being difficult to work with, O’Grady says, “Certainly in some cases if your product isn’t that popular people will abandon it.”

Cultural resistance
Another non-technological challenge to creating an open API is getting other people in the organization, who are used to dealing in proprietary information and maintaining authority over their brand, to cede some control. “I had to do a lot of convincing,” Bloomberg’s Edwards says. “It’s a different way of thinking, when you’ve been controlling your product.” But he says it was important to distinguish between the market data Bloomberg sells and things like the symbology and software that the company doesn’t need to control. “The time for all these proprietary interfaces is gone,” he says. “It doesn’t add value anymore.”

Best Buy’s Bendt also faced concerns. “It was tough when we first started talking about an API platform,” he says. Colleagues wondered, “What are they going to build? What if they create a bad experience?” The company addressed that with rules about how developers could use the data; they must attribute it to Best Buy, for instance, and can’t appropriate it for other purposes. There’s no pre-approval of apps, but the company does regular audits to make sure the apps comply with the terms of service.

At the World Bank there was worry that giving away data would mean giving up the revenue that paid for curation of the data. Fantom says the bank decided that a free model would actually be better for the bank’s main objective of fighting poverty. “By making these data available for free and using these tools, we’ve seen a massive increase in the use of our data,” he says. “Once you start getting into this, it’s pretty clear that this is the right thing to do.”

All these organizations say their APIs continue to evolve, adding new functionality, responding to feedback from developers and customers and figuring out what data to make available. “You’ve got to release the right kind of data with the right documentation. Really, it comes down to what customer problems are you going to solve by doing what you do,” Bendt says. “It’s not a launch-it-and-leave-it type of capability. It’s constant learning and constant improvement.”


TOGAF is an architecture framework. TOGAF provides the methods and tools for assisting in the acceptance, production, use, and maintenance of an enterprise architecture. It is based on an iterative process model supported by best practices and a re-usable set of existing architecture assets.

SO/IEC 42010:2007 defines “architecture” as:

“The fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.”

TOGAF embraces but does not strictly adhere to ISO/IEC 42010:2007 terminology. In TOGAF, “architecture” has two meanings depending upon the context:

  1. A formal description of a system, or a detailed plan of the system at component level to guide its implementation
  2. The structure of components, their inter-relationships, and the principles and guidelines governing their design and evolution over time

TOGAF considers the enterprise as a system and endeavors to strike a balance between promoting the concepts and terminology of ISO/IEC 42010:2007 – ensuring that usage of terms defined by ISO/IEC 42010:2007 is consistent with the standard – and retaining other commonly accepted terminology that is familiar to the majority of the TOGAF readership.

There are four architecture domains that are commonly accepted as subsets of an overall enterprise architecture, all of which TOGAF is designed to support:

  • The Business Architecture defines the business strategy, governance, organization, and key business processes.
  • The Data Architecture describes the structure of an organization’s logical and physical data assets and data management resources.
  • The Application Architecture provides a blueprint for the individual applications to be deployed, their interactions, and their relationships to the core business processes of the organization.
  • The Technology Architecture describes the logical software and hardware capabilities that are required to support the deployment of business, data, and application services. This includes IT infrastructure, middleware, networks, communications, processing, standards, etc.

Architecture Development Method

The TOGAF Architecture Development Method (ADM) provides a tested and repeatable process for developing architectures. The ADM includes establishing an architecture framework, developing architecture content, transitioning, and governing the realization of architectures.

All of these activities are carried out within an iterative cycle of continuous architecture definition and realization that allows organizations to transform their enterprises in a controlled manner in response to business goals and opportunities.

Phases within the ADM are as follows:

  • The Preliminary Phase describes the preparation and initiation activities required to create an Architecture Capability including customization of TOGAF and definition of Architecture Principles.
  • Phase A: Architecture Vision describes the initial phase of an architecture development cycle. It includes information about defining the scope of the architecture development initiative, identifying the stakeholders, creating the Architecture Vision, and obtaining approval to proceed with the architecture development.
  • Phase B: Business Architecture describes the development of a Business Architecture to support the agreed Architecture Vision.
  • Phase C: Information Systems Architectures describes the development of Information Systems Architectures to support the agreed Architecture Vision.
  • Phase D: Technology Architecture describes the development of the Technology Architecture to support the agreed Architecture Vision.
  • Phase E: Opportunities & Solutions conducts initial implementation planning and the identification of delivery vehicles for the architecture defined in the previous phases.
  • Phase F: Migration Planning addresses how to move from the Baseline to the Target Architectures by finalizing a detailed Implementation and Migration Plan.
  • Phase G: Implementation Governance provides an architectural oversight of the implementation.
  • Phase H: Architecture Change Management establishes procedures for managing change to the new architecture.
  • Requirements Management examines the process of managing architecture requirements throughout the ADM.

Deliverables, Artifacts, and Building Blocks

Architects executing the ADM will produce a number of outputs as a result of their efforts, such as process flows, architectural requirements, project plans, project compliance assessments, etc. The TOGAF Architecture Content Framework   provides a structural model for architectural content that allows major work products to be consistently defined, structured, and presented.

The Architecture Content Framework uses the following three categories to describe the type of architectural work product within the context of use:

  • A deliverable is a work product that is contractually specified and in turn formally reviewed, agreed, and signed off by the stakeholders. Deliverables represent the output of projects and those deliverables that are in documentation form will typically be archived at completion of a project, or transitioned into an Architecture Repository as a reference model, standard, or snapshot of the Architecture Landscape at a point in time.
  • An artifact is an architectural work product that describes an aspect of the architecture. Artifacts are generally classified as catalogs (lists of things), matrices (showing relationships between things), and diagrams (pictures of things). Examples include a requirements catalog, business interaction matrix, and a use-case diagram. An architectural deliverable may contain many artifacts and artifacts will form the content of the Architecture Repository.
  • A building block represents a (potentially re-usable) component of business, IT, or architectural capability that can be combined with other building blocks to deliver architectures and solutions.Building blocks can be defined at various levels of detail, depending on what stage of architecture development has been reached. For instance, at an early stage, a building block can simply consist of a name or an outline description. Later on, a building block may be decomposed into multiple supporting building blocks and may be accompanied by a full specification. Building blocks can relate to “architectures” or “solutions”.
    • Architecture Building Blocks (ABBs) typically describe required capability and shape the specification of Solution Building Blocks (SBBs). For example, a customer services capability may be required within an enterprise, supported by many SBBs, such as processes, data, and application software.
    • Solution Building Blocks (SBBs) represent components that will be used to implement the required capability. For example, a network is a building block that can be described through complementary artifacts and then put to use to realize solutions for the enterprise.

The relationships between deliverables, artifacts, and building blocks are shown in Figure 1.

Figure -1: Relationships between Deliverables, Artifacts, and Building Blocks

For example, an Architecture Definition Document is a deliverable that documents an architecture description. This document will contain a number of complementary artifacts that are views of the building blocks relevant to the architecture. For example, a process flow diagram (an artifact) may be created to describe the target call handling process (a building block). This artifact may also describe other building blocks, such as the actors involved in the process (e.g., a Customer Services Representative). An example of the relationships between deliverables, artifacts, and building blocks is illustrated in Figure 2

Figure 2: Example – Architecture Definition Document

Enterprise Continuum

TOGAF includes the concept of the Enterprise Continuum, which sets the broader context for an architect and explains how generic solutions can be leveraged and specialized in order to support the requirements of an individual organization. The Enterprise Continuum is a view of the Architecture Repository that provides methods for classifying architecture and solution artifacts as they evolve from generic Foundation Architectures to Organization-Specific Architectures. The Enterprise Continuum comprises two complementary concepts: the Architecture Continuum and the Solutions Continuum.

An overview of the structure and context for the Enterprise Continuum is shown in Figure 3.

Figure -3: Enterprise Continuum

Architecture Repository

Supporting the Enterprise Continuum is the concept of an Architecture Repository which can be used to store different classes of architectural output at different levels of abstraction, created by the ADM. In this way, TOGAF facilitates understanding and co-operation between stakeholders and practitioners at different levels.

By means of the Enterprise Continuum and Architecture Repository, architects are encouraged to leverage all other relevant architectural resources and assets in developing an Organization-Specific Architecture.

In this context, the TOGAF ADM can be regarded as describing a process lifecycle that operates at multiple levels within the organization, operating within a holistic governance framework and producing aligned outputs that reside in an Architecture Repository. The Enterprise Continuum provides a valuable context for understanding architectural models: it shows building blocks and their relationships to each other, and the constraints and requirements on a cycle of architecture development.

The structure of the TOGAF Architecture Repository is shown in Figure  4.

Figure 4: TOGAF Architecture Repository StructureThe major components within an Architecture Repository are as follows:

  • The Architecture Metamodel describes the organizationally tailored application of an architecture framework, including a metamodel for architecture content.
  • The Architecture Capability defines the parameters, structures, and processes that support governance of the Architecture Repository.
  • The Architecture Landscape is the architectural representation of assets deployed within the operating enterprise at a particular point in time. The landscape is likely to exist at multiple levels of abstraction to suit different architecture objectives.
  • The Standards Information Base (SIB) captures the standards with which new architectures must comply, which may include industry standards, selected products and services from suppliers, or shared services already deployed within the organization.
  • The Reference Library provides guidelines, templates, patterns, and other forms of reference material that can be leveraged in order to accelerate the creation of new architectures for the enterprise.
  • The Governance Log provides a record of governance activity across the enterprise.

Establishing and Maintaining an Enterprise Architecture Capability

In order to carry out architectural activity effectively within an enterprise, it is necessary to put in place an appropriate business capability for architecture, through organization structures, roles, responsibilities, skills, and processes. An overview of the TOGAF Architecture Capability is shown in Figure 5.

Figure 5: TOGAF Architecture Capability Overview

Establishing the Architecture Capability as an Operational Entity

Barring architecture capabilities set up to purely support change delivery programs, it is increasingly recognized that a successful enterprise architecture practice must sit on a firm operational footing. In effect, an enterprise architecture practice must be run like any other operational unit within a business; i.e., it should be treated like a business. To this end, and over and above the core processes defined within the ADM, an enterprise architecture practice should establish capabilities in the following areas:

  • Financial Management
  • Performance Management
  • Service Management
  • Risk Management
  • Resource Management
  • Communications and Stakeholder Management
  • Quality Management
  • Supplier Management
  • Configuration Management
  • Environment Management

Central to the notion of operating an ongoing architecture is the execution of well-defined and effective governance, whereby all architecturally significant activity is controlled and aligned within a single framework.

As governance has become an increasingly visible requirement for organizational management, the inclusion of governance within TOGAF aligns the framework with current business best practice and also ensures a level of visibility, guidance, and control that will support all architecture stakeholder requirements and obligations.

The benefits of architecture governance include:

  • Increased transparency of accountability, and informed delegation of authority
  • Controlled risk management
  • Protection of the existing asset base through maximizing re-use of existing architectural components
  • Proactive control, monitoring, and management mechanisms
  • Process, concept, and component re-use across all organizational business units
  • Value creation through monitoring, measuring, evaluation, and feedback
  • Increased visibility supporting internal processes and external parties’ requirements; in particular, increased visibility of decision-making at lower levels ensures oversight at an appropriate level within the enterprise of decisions that may have far-reaching strategic consequences for the organization
  • Greater shareholder value; in particular, enterprise architecture increasingly represents the core intellectual property of the enterprise – studies have demonstrated a correlation between increased shareholder value and well-governed enterprises
  • Integrates with existing processes and methodologies and complements functionality by adding control capabilities

Using TOGAF with Other Frameworks

Two of the key elements of any enterprise architecture framework are:

  • A definition of the deliverables that the architecting activity should produce
  • A description of the method by which this should be done

With some exceptions, the majority of enterprise architecture frameworks focus on the first of these – the specific set of deliverables – and are relatively silent about the methods to be used to generate them (intentionally so, in some cases).

Because TOGAF is a generic framework and intended to be used in a wide variety of environments, it provides a flexible and extensible content framework that underpins a set of generic architecture deliverables.

As a result, TOGAF may be used either in its own right, with the generic deliverables that it describes; or else these deliverables may be replaced or extended by a more specific set, defined in any other framework that the architect considers relevant.

In all cases, it is expected that the architect will adapt and build on the TOGAF framework in order to define a tailored method that is integrated into the processes and organization structures of the enterprise. This architecture tailoring may include adopting elements from other architecture frameworks, or integrating TOGAF methods with other standard frameworks, such as ITIL, CMMI, COBIT, PRINCE2, PMBOK, and MSP. Guidelines for adapting the TOGAF ADM

As a generic framework and method for enterprise architecture, TOGAF provides the capability and the collaborative environment to integrate with other frameworks. Organizations are able to fully utilize vertical business domains, horizontal technology areas (such as security or manageability), or application areas (such as e-Commerce) to produce a competitive enterprise architecture framework which maximizes their business opportunities.


COBIT 5 is the only business framework for the governance and management of enterprise IT. This evolutionary version incorporates the latest thinking in enterprise governance and management techniques, and provides globally accepted principles, practices, analytical tools and models to help increase the trust in, and value from, information systems. COBIT 5 builds and expands on COBIT 4.1 by integrating other major frameworks, standards and resources, including ISACA’s Val IT and Risk IT, Information Technology Infrastructure Library (ITIL®) and related standards from the International Organization for Standardization (ISO).

Control Objectives for Information and Related Technology (COBIT) is a framework created by ISACA for information technology (IT) management and IT governance. It is a supporting toolset that allows managers to bridge the gap between control requirements, technical issues and business risks.


COBIT was first released in 1996; the current version, COBIT 5, was published in 2012. Its mission is “to research, develop, publish and promote an authoritative, up-to-date, international set of generally accepted information technology control objectives for day-to-day use by business managers, IT professionals and assurance professionals.”.

COBIT, initially an acronym for ‘Control objectives for information and related technology’, defines a set of generic processes to manage IT. Each process is defined together with process inputs and outputs, key process activities, process objectives, performance measures and an elementary maturity model. The framework supports governance of IT by defining and aligning business goals with IT goals and IT processes.

The COBIT framework

The framework provides good practices across a domain and process framework.

The business orientation of COBIT consists of linking business goals to IT goals, providing metrics and maturity models to measure their achievement, and identifying the associated responsibilities of business and IT process owners.

The process focus of COBIT 4.1 is illustrated by a process model that subdivides IT into four domains (Plan and Organize, Acquire and Implement, Deliver and Support, and Monitor and Evaluate) and 34 processes in line with the responsibility areas of plan, build, run and monitor. It is positioned at a high level and has been aligned and harmonized with other, more detailed, IT standards and good practices such as COSO, ITIL, ISO 27000, CMMI, TOGAF and PMBOK. COBIT acts as an integrator of these different guidance materials, summarizing key objectives under one umbrella framework that link the good practice models with governance and business requirements.

The COBIT 4.1 framework specification can be obtained as a complimentary PDF at the ISACA download website. (Free self-registration may be required.)

COBIT 5 was released in June 2012.COBIT 5 consolidates and integrates the COBIT 4.1, Val IT 2.0 and Risk IT frameworks, and draws from ISACA’s IT Assurance Framework (ITAF) and the Business Model for Information Security (BMIS). It aligns with frameworks and standards such as Information Technology Infrastructure Library (ITIL), International Organization for Standardization (ISO), Project Management Body of Knowledge (PMBOK), PRINCE2 and The Open Group Architecture Framework (TOGAF).


COBIT has had five major releases:

  • In 1996, the first edition of COBIT was released.
  • In 1998, the second edition added “Management Guidelines”.
  • In 2000, the third edition was released.
    • In 2003, an on-line version became available.
  • In December 2005, the fourth edition was initially released.
    • In May 2007, the current 4.1 revision was released.
  • COBIT 5 was released in June 2012. It consolidates and integrates the COBIT 4.1, Val IT 2.0 and Risk IT frameworks, and also draws significantly from the Business Model for Information Security (BMIS) and ITAF.


The COBIT components include:

  • Framework: Organize IT governance objectives and good practices by IT domains and processes, and links them to business requirements
  • Process descriptions: A reference process model and common language for everyone in an organization. The processes map to responsibility areas of plan, build, run and monitor.
  • Control objectives: Provide a complete set of high-level requirements to be considered by management for effective control of each IT process.
  • Management guidelines: Help assign responsibility, agree on objectives, measure performance, and illustrate interrelationship with other processes
  • Maturity models: Assess maturity and capability per process and helps to address gaps.

Other ISACA Publications based on the COBIT framework include:

  • Board Briefing for IT Governances, 2nd Edition
  • COBIT and Application Controls
  • COBIT Control Practices, 2nd Edition
  • IT Assurance Guide: Using COBIT
  • Implementing and Continually Improving IT Governance
  • COBIT Quickstart, 2nd Edition
  • COBIT Security Baseline, 2nd Edition
  • IT Control Objectives for Sarbanes-Oxley, 2nd Edition
  • IT Control Objectives for Basel II
  • COBIT User Guide for Service Managers
  • COBIT Mappings (to ISO/IEC 27002, CMMI, ITIL, TOGAF, PMBOK etc.)
  • COBIT Online

COBIT and Sarbanes-Oxley

Companies that are publicly traded in the US are subject to the Sarbanes-Oxley Act of 2002. According to the IIA, COBIT is one of the most commonly used frameworks to comply with Sarbanes-Oxley.


COBIT 5 helps enterprises of all sizes:

  • Maintain high-quality information to support business decisions
  • Achieve strategic goals and realize business benefits through the effective and innovative use of IT
  • Achieve operational excellence through reliable, efficient application of technology
  • Maintain IT-related risk at an acceptable level
  • Optimize the cost of IT services and technology
  • Support compliance with relevant laws, regulations, contractual agreements and policies

Simply stated, COBIT 5 helps enterprises create optimal value from IT by maintaining a balance between realising benefits and optimising risk levels and resource use.

COBIT 5 enables information and related technology to be governed and managed in a holistic manner for the entire enterprise, taking in the full end-to-end business and functional areas of responsibility, considering the IT-related interests of internal and external stakeholders.

The COBIT 5 principles and enablers are generic and useful for enterprises of all sizes, whether commercial, not-for-profit or in the public sector.

Governance ensures that enterprise objectives are achieved by evaluating stakeholder needs, conditions and options; setting direction through prioritisation and decision making; and monitoring performance, compliance and progress against agreed-on direction and objectives (EDM).

Managementplans, builds, runs and monitors activities in alignment with the direction set by the governance body to achieve the enterprise objectives (PBRM).

The five COBIT 5 principles:

1.Meeting Stakeholder Needs

2.Covering the Enterprise End-to-end

3.Applying a Single Integrated Framework

4.Enabling a Holistic Approach

5.Separating Governance From Management


COBIT is a framework developed for IT process management with a strong focus on control. These scales need to be practical to apply and reasonably easy to understand. The topic of IT process management is inherently complex and subjective and, therefore, is best approached through facilitated assessments that raise awareness, capture broad consensus and motivate improvement. These assessments can be performed either against the maturity level descriptions as a whole or with more rigour against each of the individual statements of the descriptions. Either way, expertise in the enterprise’s process under review is required.

The advantage of a maturity model approach is that it is relatively easy for management to place itself on the scale and appreciate what is involved if improved performance is needed. The scale includes 0 because it is quite possible that no process exists at all. The 0-5 scale is based on a simple maturity scale showing how a process evolves from a non-existent capability to an optimised capability.

However, process management capability is not the same as process performance. The required capability, as determined by business and IT goals, may not need to be applied to the same level across the entire IT environment, e.g., not consistently or to only a limited number of systems or units. Performance measurement,  is essential in determining what the enterprise’s actual performance is for its IT processes.

Although a properly applied capability already reduces risks, an enterprise still needs to analyse the controls necessary to ensure that risk is mitigated and value is obtained in line with the risk appetite and business objectives. These controls are guided by COBIT’s control objectives.

Although it is difficult to imagine today, there was a time in the not-too-distant past the term “governance” was not used so often or so freely. Based on Latin and Greek words referring to the steering of a ship, governance was a concept primarily restricted to the CEO’s office and the boardroom.

But several global business catastrophes over the last few decades brought the term to the forefront of business thinking. Massive trader fraud at Barings Bank and Société Générale uncovered a breakdown in the monitoring of internal processes. Enron’s phenomenal success was revealed to be due, in some measure, to systematic, long-term, organized accounting fraud. WorldCom, Global Crossing and Tyco followed, and soon there was a clear and widely accepted need for more rigorous governance over companies’ systems of internal control. Increasingly, legislation is being passed to address this need. Worldwide, regulations such as Basel II, the Canadian Privacy Act, HIPAA, Australia’s Corporate Law Economic Reform Program and the Sarbanes-Oxley Act have moved governance to the top of agendas at all layers within enterprises.

This demonstrates that, although the highly publicized enterprise meltdowns may have focused more attention on formalized systems of governance, considerable thought and exploration had already gone into the basic concepts. As stakeholder issues, corporate social responsibility, enterprise risk management and alliances became an increasingly critical component of enterprise success, many astute enterprise leaders realized the need for effective and efficient governance around these issues.

The various legislative acts mentioned above may have reinforced the already existing focus on governance, but they did not necessarily bring clarity to the topic. Different views of governance have arisen, leading to confusion on the relationships among them. How does corporate governance relate to enterprise governance? What about the differing views brought about by increased dependency on intellectual capital, people, information technology—isn’t there a need for governance of these as well? What governance principles apply to open collaboration (“wiki”) sites, whose multiple authors and shared intellectual property create muddied ownership issues? As governance issues have become increasingly complex and critical to the business, those who need to understand and apply these concepts often find themselves wishing “if only someone would put all this on a page, so I could see how it fits together.”

Governance—Beyond Compliance
Although compliance is a powerful driver, there are more profound reasons to implement effective and efficient governance. Enterprise complexity, corporate responsibility, transparency—all these facets of the current business environment bring challenges that governance can address. Among the other drivers for efficient and effective governance:

  • Transparency, as stakeholders wish to be aware of the decisions, mechanisms and results of the enterprises in which they have an interest
  • Enterprises’ need to practice and demonstrate corporate responsibility
  • Creation of the extended enterprise, as enterprises’ growing appreciation of the benefits of collaboration has led to an increasingly large and complex network of relationships and strategic partners (joint ventures, wiki partners, etc.), among which delegation of accountability and decision-making rights  must be accomplished
  • Multiple stakeholders’ varying understanding and acceptance of value and risk
  • Enterprises’ increased dependence on strategic assets (e.g., people, knowledge, communications infrastructure, energy infrastructure, intellectual capital, information technology, information) or on external parties (e.g., outsourcing, strategic alliances) and the critical need for 24/7 access to them.
  • Data and information are especially important assets, as they form the basis for informed business decisions; bad data can lead to bad decisions.

The end result is an increasingly complex web, characterized by multiple layers of detail, requirements and interdependencies. This initiative has been undertaken to help clarify and provide ways to maneuver through the governance maze.

The governance mapping initiative was undertaken for many reasons. In addition to gaining a better understanding of the governance space and how the many components fit together, the intent is to populate the resulting maps in ways that will help all organizations dealing with these topics gain a better comprehension of what and who are operating in the governance space. Ultimately, it is hoped that the final product will encourage and assist enterprises to apply effective and efficient governance concepts within their own structures.

The mapping will therefore provide enterprises with guidance on existing standards, frameworks and methodologies that can assist them in implementing effective governance quickly and efficiently, and that incorporate the best practices appropriate to them. Information plays a vital role in the space and fosters good communication among related parties for better governance.

Therefore, the long-term plan is to populate the map—starting with the version that features IT governance (since that is ITGI’s area of expertise)—with four basic types of information:

  1. The leading international guidance, standards and frameworks relating to the disciplines reflected on the map. It is not intended to include all the guidance documents that exist, at this stage. The first iterations of the map will focus on what are considered the “top” guidance documents.Many frameworks can apply to many areas on the map; this map places them where their primary focus lies.
    In addition to showing just the “alphabet soup” of acronyms, each listing will be linked to further information describing the issuing body, the general purpose of the guidance, and the ways in which it specifically addresses the space in the governance map to which it has been assigned. ITGI’s worldwide cadre of experts will make inroads into this, but ultimately external assistance will be requested.
  2. The people involved in each view of governance—specifically the roles, responsibilities, credentials and qualifications of each
  3. The organizations that offer complementary products or services in each area
  4. ITGI’s own products and service offerings. Other organizations may wish to do the same for their own portfolio.

The mapping and description of the relevant frameworks, standards and methodologies will provide enterprises with practical guidance on governance implementation by clarifying aspects such as:

  • The scope of the framework, standard or methodology (e.g., COBIT’s positioning as a comprehensive process and control framework covering all IT processes vis-à-vis The Open Group Architecture Framework’s [TOGAF] positioning within the enterprise architecture domain)
  • Particular suitability to specific industries (where these exist)
  • Expected benefits from the application of the relevant framework, standard or methodology
  • Current uptake and reach (e.g., global reach for Committee of Sponsoring Organizations of the Treadway Commission [COSO])

What the new TLDs mean for brands and consumers

Domain changes afoot on the internet and what we can expect to see from this new online landscape in the coming months

In 2011, ICANN, The Internet Corporation for Assigned Names and Numbers (ICANN) the body that governs addresses on the internet, recently made the decision to allow brands and organisations to apply for and manage their own unique top-level domains (TLD). Businesses (and others) that submitted a successful application and met all of ICANN’s requirements are about to have their name featured on the right side of the dot in web addresses for the first time. In ICANN’s ‘dot Brand’ new world, you can expect to see domains such as .CHANEL or .NIKE alongside familiar domains such as .COM, .INFO and .ORG. In short, brands are no longer constrained to the left of the dot (as in ‘nike.com’), but can now have their own identity at the very top level (as in ‘sports.nike’).

Dawn of a new domain

The internet’s makeover will begin to show up publicly later this year; that’s when we expect to see the first of hundreds of new domains that will be launched. ICANN received almost 2,000 applications – some for branded domains such as .HSBC, but also applications for generic terms like .FOOD and for geographic regions like .LONDON and .RIO.

The decision to expand the TLD system was made to introduce greater internet innovation and competition. One key innovation is to enable brands to free themselves from the need to market under irrelevant names like ‘.com’ or ‘Facebook’, and focus entirely on their own identity. Since domain names are now a critical component of a brand’s identity, ICANN’s gateway to TLD ownership represents a turning point in brand management on the internet. When the new domains launch later this year, we’ll certainly see revised marketing strategies from these brands, who will be keen to reap the advantages such domains offer, including a strengthened online identity, improved security and new possibilities for customer engagement.

However, dot Brands also present benefits to online consumers. In particular, these domains will inject greater confidence into the ecommerce space, an area that has been plagued with counterfeiting and other security issues. By ICANN mandate, all new TLDs must be DNSSEC enabled. This relatively security new technology helps ensure that consumers who type in an address with a dot Brand extension will arrive at the intended site, instead of a phishing or other counterfeit-type site. Especially for shoppers of luxury goods or pharmaceuticals, dot Brand addresses finally enable confidence that the goods are from a legitimate source. Furthermore, since each dot Brand is completely controlled by the brand owner such as Rolex, they can exercise total control over who is permitted to use a .ROLEX web address.

Another key innovation is the introduction of non-Roman scripts for top-level domains. This will enable Chinese character addresses, as well as addresses in Cyrillic, Korean, Japanese, Hindi and so on … thus opening up the internet to millions of new users.

Without the expansion of top level domains, many potential participants in the global internet may have continued to find themselves excluded. The multi-language internet is critical to bridging the digital divide, particularly when internet availability expands to non-English speaking areas. For global retailers and marketers, non-Roman script names present the opportunity to connect with the world’s growing markets in local languages like Arabic, Russian and Chinese. Amazon is one of the first to move into this space with its application for 家電 (Japanese script for “consumer electronics”) while L’Oréal has opted for 欧莱雅 (the Chinese transliteration of “L’Oréal”).

Dot foes?

Despite the benefits, the prospect of dot Brand top-level domains have generated controversy and disagreement. Some opponents disagree with businesses being allowed to apply for generic terms such as .BEAUTY, while religious and conservative groups object to domains such as .ISLAM or .SEXY. Others have voiced concerns about players like Amazon and Google reinforcing their dominant market positions and monopolising entire online categories as with the respective applications for .BOOKS and .SHOP.

However, these terms exist now at the second level in many existing domains (for example, beauty.com, beauty.info) and have not caused any particular problems. Further, generic terms have been trademarked in various jurisdictions for decades. Given these facts, the opportunity to have them at the top level of a domain name seems to be a natural evolution.

Standing out in a dot Brand crowd

Although the initial application period closed last year, ICANN has said that the opportunity to own a TLD will come again in the future once the first batch of dot Brands are delegated and active. Even if you missed the first round, you can put other proactive measures in place to help ensure your brand is not overlooked in the new, branded online landscape.

First, businesses should ensure that its main website address is the central location for all digital assets. Diverting an online audience to other channels, whether it’s a Facebook page or Twitter handle, effectively reroutes them to another company and dilutes a business’s own online brand presence. While a social media presence is essential, every brand needs its own unique home that can be completely controlled by the company’s brand management and where customers cannot be tracked or otherwise exploited by a third party.

Additionally, with Forrester predicting that mobile commerce is set to quadruple to $31 billion in the next five years, preparing for the m-commerce boom is essential for competing in the new online environment. Consumers now expect a seamless online experience, regardless of the type of device used to access a business’s website. To that end, businesses must ensure that existing websites are optimised for mobile viewing.

While the success of the new TLDs is yet to be determined, one thing is for sure: the internet is continuing to grow and play an ever-larger role in our lives. Regardless of whether a business has applied for a dot Brand, the tide of change sweeping across the internet means businesses need to ensure their digital presence is in order in 2013 and beyond.

ITIL – Information Technology Infrastructure Library

The Information Technology Infrastructure Library (ITIL) is a set of practices for IT service management (ITSM) that focuses on aligning IT services with the needs of business. In its current form (known as ITIL 2011 edition), ITIL is published in a series of five core publications, each of which covers an ITSM lifecycle stage. ITIL underpins ISO/IEC 20000 (previously BS15000), the International Service Management Standard for IT service management, although differences between the two frameworks do exist.

ITIL describes processes, procedures, tasks and checklists that are not organization-specific, used by an organization for establishing integration with the organization’s strategy, delivering value and maintaining a minimum level of competency. It allows the organization to establish a baseline from which it can plan, implement, and measure. It is used to demonstrate compliance and to measure improvement.

The acronym ITIL is a registered trademark of the United Kingdom’s Cabinet Office. Following this move, the ownership is now listed as being with HM Government rather than OGC. The publications continue to be Crown Copyright.


Responding to growing dependence on IT, the UK Government’s Central Computer and Telecommunications Agency (CCTA) in the 1980s developed a set of recommendations. It recognised that without standard practices, government agencies and private sector contracts had started independently creating their own IT management practices.

The IT Infrastructure Library originated as a collection of books, each covering a specific practice within IT service management. ITIL was built around a process-model based view of controlling and managing operations often credited to W. Edwards Deming and his plan-do-check-act (PDCA) cycle.

After the initial publication in 1989–96, the number of books quickly grew within ITIL v1 to more than 30 volumes.

In 2000/2001, to make ITIL more accessible (and affordable), ITIL v2 consolidated the publications into eight logical “sets” that grouped related process-guidelines to match different aspects of IT management, applications, and services. The Service Management sets (Service Support and Service Delivery) were by far the most widely used, circulated, and understood[citation needed] of ITIL v2 publications.

    In April 2001 the CCTA was merged into the Office of Government Commerce (OGC), an office of the UK Treasury.

    In 2006, the ITIL v2 glossary was published.

    In May 2007, this organisation issued version 3 of ITIL (also known as the ITIL Refresh Project) consisting of 26 processes and functions, now grouped into only 5 volumes, arranged around the concept of Service lifecycle structure. Version 3 is now known as ITIL 2007 Edition.

    In 2009, the OGC officially announced that ITIL v2 certification would be withdrawn and launched a major consultation as per how to proceed.

    In July 2011, the 2011 edition of ITIL was published, providing an update to the version published in 2007. The OGC is no longer listed as the owner of ITIL, following the consolidation of OGC into the Cabinet Office. The 2011 edition is owned by HM Government.

Overview of ITIL 2007 Edition

ITIL 2007 Edition (previously known as version 3) is an extension of ITIL v2 and fully replaced it following the completion of the withdrawal period on 30 June 2011.[4] ITIL 2007 provides a more holistic perspective on the full life cycle of services, covering the entire IT organisation and all supporting components needed to deliver services to the customer, whereas v2 focused on specific activities directly related to service delivery and support. Most of the v2 activities remained untouched in 2007, but some significant changes in terminology were introduced in order to facilitate the expansion.

Changes and characteristics of the 2011 edition of ITIL

A summary of changes has been published by HM Government. In line with the 2007 edition, the 2011 edition consists of five core publications – Service Strategy, Service Design, Service Transition, Service Operation, and Continual Service Improvement. ITIL 2011 is an update to the ITIL framework that addresses significant additional guidance with the definition of formal processes which were previously implied but not identified, as well as correction of errors and inconsistencies.

There are twenty-six processes listed in ITIL 2011 edition and described below that shows which core publication provides the main content for each process.

ITIL 2007 has five volumes, published in May 2007 and updated in July 2011 as ITIL 2011 for consistency:

    ITIL Service Strategy

    ITIL Service Design

    ITIL Service Transition

    ITIL Service Operation

    ITIL Continual Service Improvement

Service strategy

As the centre and origin point of the ITIL Service Lifecycle, the ITIL Service Strategy (SS) volume provides guidance on clarification and prioritization of service-provider investments in services. More generally, Service Strategy focuses on helping IT organizations improve and develop over the long term. In both cases, Service Strategy relies largely upon a market-driven approach. Key topics covered include service value definition, business-case development, service assets, market analysis, and service provider types.

List of covered processes:

  •     IT service management
  •     Service Portfolio Management
  •     Financial management for IT services
  •     Demand Management
  •     Business relationship management

For candidates in the ITIL Intermediate Capability stream, the Service Offerings and Agreements (SOA) Qualification course and exam are most closely aligned to the Service Strategy (SS) Qualification course and exam in the Lifecycle stream.

Financial management for IT services

IT Financial Management comprises the discipline of ensuring that the IT infrastructure is obtained at the most effective price (which does not necessarily mean cheapest) and calculating the cost of providing IT services so that an organization can understand the costs of its IT services. These costs may then be recovered from the customer of the service. This is the 2nd component of service delivery process.

Service design

The Service Design (SD) volume provides good-practice guidance on the design of IT services, processes, and other aspects of the service management effort. Significantly, design within ITIL is understood to encompass all elements relevant to technology service delivery, rather than focusing solely on design of the technology itself. As such, service design addresses how a planned service solution interacts with the larger business and technical environments, service management systems required to support the service, processes which interact with the service, technology, and architecture required to support the service, and the supply chain required to support the planned service. Within ITIL, design work for an IT service is aggregated into a single service design package (SDP). Service design packages, along with other information about services, are managed within the service catalogues.

List of covered processes:

    Design coordination (Introduced in ITIL 2011 Edition)

    Service Catalogue

    Service level Management

    Availability Management

    Capacity Management

    IT Service Continuity Management (ITSCM)

    Information Security Management System

    Supplier Management

Service-level management

Service-level management provides for continual identification, monitoring and review of the levels of IT services specified in the Service-level agreements (SLAS). Service-level management ensures that arrangements are in place with internal IT support-providers and external suppliers in the form of Operational Level Agreements OLAS and Underpinning Contracts (UCLS), respectively. The process involves assessing the impact of change upon service quality and SLAS. The service-level management process is in close relation with the operational processes to control their activities. The central role of Service-level management makes it the natural place for metrics to be established and monitored against a benchmark.

Service-level management is the primary interface with the customer (as opposed to the user serviced by the service desk). Service-level management is responsible for:

    Ensuring that the agreed IT services are delivered when and where they are supposed to be

    Liaising with availability management, capacity management, incident management and problem management to ensure that the required levels and quality of service are achieved within the resources agreed with financial management

    Producing and maintaining a service catalog (a list of standard IT service options and agreements made available to customers)

    Ensuring that appropriate IT service continuity plans exist to support the business and its continuity requirements.

The service-level manager relies on the other areas of the service delivery process to provide the necessary support which ensures the agreed services are provided in a cost-effective, secure and efficient manner.

Availability management

Availability management targets allowing organizations to sustain the IT service-availability to support the business at a justifiable cost. The high-level activities realize availability requirements, compile availability plan, monitor availability, and monitor maintenance obligations.

Availability management addresses the ability of an IT component to perform at an agreed level over a period of time.

    Reliability: Ability of an IT component to perform at an agreed level at described conditions.

    Maintainability: The ability of an IT component to remain in, or be restored to an operational state.

    Serviceability: The ability for an external supplier to maintain the availability of component or function under a third-party contract.

    Resilience: A measure of freedom from operational failure and a method of keeping services reliable. One popular method of resilience is redundancy.

    Security: A service may have associated data. Security refers to the confidentiality, integrity, and availability of that data. Availability gives a clear overview of the end-to-end availability of the system.

Capacity management

Capacity management supports the optimum and cost-effective provision of IT services by helping organizations match their IT resources to business demands. The high-level activities include:

    Application Sizing

    Workload Management

    Demand Management


    Capacity Planning

    Resource Management

    Performance Management

Capacity management is focused on strategic capacity, including capacity of personnel (e.g., human resources, staffing and training), system capacity, and component (or tactical) capacity.

IT service continuity management

IT service continuity management (ITSCM) covers the processes by which plans are put in place and managed to ensure that IT Services can recover and continue even after a serious incident occurs. It is not just about reactive measures, but also about proactive measures – reducing the risk of a disaster in the first instance.

ITSCM is regarded by the application owners as the recovery of the IT infrastructure used to deliver IT Services, but as of 2009 many businesses practice the much further-reaching process of business continuity planning (BCP), to ensure that the whole end-to-end business process can continue should a serious incident occur (at primary support level).

ITSCM involves the following basic steps:

    Prioritising the activities to be recovered by conducting a business impact analysis (BIA)

    Performing a risk assessment (aka risk analysis) for each of the IT services to identify the assets, threats, vulnerabilities and countermeasures for each service.

    Evaluating the options for recovery

    Producing the contingency plan

    Testing, reviewing, and revising the plan on a regular basis.

Information security management system

The ITIL-process Security Management  describes the structured fitting of information security in the management organisation. ITIL security management is based on the code of practice for information security management system (ISMS) now known as ISO/IEC 27002.

A basic goal of security management is to ensure adequate information security. The primary goal of information security, in turn, is to protect information assets against risks, and thus to maintain their value to the organization. This is commonly expressed in terms of ensuring their confidentiality, integrity and availability, along with related properties or goals such as authenticity, accountability, non-repudiation and reliability.

Mounting pressure for many organisations to structure their information security management systems in accordance with ISO/IEC 27001 requires revision of the ITIL v2 security management volume, which culminated in the release of the 2007 edition.

Service transition

Service transition (ST), as described by the ITIL service transition volume,[7] relates to the delivery of services required by a business into live/operational use, and often encompasses the “project” side of IT rather than business as usual (BAU). This area also covers topics such as managing changes to the BAU environment.

List of ITIL processes in service transition:

    Transition planning and support

    Change management

    Service asset and configuration management

    Release and deployment management

    Service validation and testing

    Change evaluation

    Knowledge management

Change management

Change management aims to ensure that standardised methods and procedures are used for efficient handling of all changes. A change is an event that results in a new status of one or more configuration items (CIS), and which is approved by management, cost-effective, enhances business process changes (fixes) – all with a minimum risk to IT infrastructure.

The main aims of change management include:

    Minimal disruption of services

    Reduction in back-out activities

    Economic use of resources involved in the change

Common change management terminology includes:

    Change: the addition, modification or removal of CIS

    Request For Change (RFC) or, in older terminology, Change Request (CR): a form used to record details of a request for a change and is sent as an input to Change Management by the Change Requestor

    ITIL v2 – Forward Schedule of Changes (FSC): schedule that contains details of all forthcoming Changes.

    ITIL 2007 – Change Schedule (CS): schedule that contains details of all forthcoming Changes, and references historical data. Many people still refer to the known term FSC.

Service asset and configuration management

Service asset and configuration management is primarily focused on maintaining information (i.e., configurations) about Configuration Items (i.e., assets) required to deliver an IT service, including their relationships. Configuration management is the management and traceability of every aspect of a configuration from beginning to end and it includes the following key process areas under its umbrella:



    Change Control,

    Change Management,

    Release Management, and


Release and deployment management

Release and deployment management is used by the software migration team for platform-independent and automated distribution of software and hardware, including license controls across the entire IT infrastructure. Proper software and hardware control ensures the availability of licensed, tested, and version-certified software and hardware, which functions as intended when introduced into existing infrastructure. Quality control during the development and implementation of new hardware and software is also the responsibility of Release Management. This guarantees that all software meets the demands of the business processes.

The goals of release management include:

    Planning the rollout of software

    Designing and implementing procedures for the distribution and installation of changes to IT systems

    Effectively communicating and managing expectations of the customer during the planning and rollout of new releases

    Controlling the distribution and installation of changes to IT systems

Release management focuses on the protection of the live environment and its services through the use of formal procedures and checks.

A Release consists of the new or changed software and/or hardware required to implement approved changes. Release categories include:

    Major software releases and major hardware upgrades, normally containing large amounts of new functionality, some of which may make intervening fixes to problems redundant. A major upgrade or release usually supersedes all preceding minor upgrades, releases and emergency fixes.

    Minor software releases and hardware upgrades, normally containing small enhancements and fixes, some of which may have already been issued as emergency fixes. A minor upgrade or release usually supersedes all preceding emergency fixes.

    Emergency software and hardware fixes, normally containing the corrections to a small number of known problems.

Releases can be divided based on the release unit into:

    Delta release: a release of only that part of the software which has been changed. For example, security patches.

    Full release: the entire software program is deployed—for example, a new version of an existing application.

    Packaged release: a combination of many changes—for example, an operating system image which also contains specific applications.

Service operation

Service Operation (SO) aims to provide best practice for achieving the delivery of agreed levels of services both to end-users and the customers (where “customers” refer to those individuals who pay for the service and negotiate the slas). Service operation, as described in the ITIL Service Operation volume, is the part of the lifecycle where the services and value is actually directly delivered. Also the monitoring of problems and balance between service reliability and cost etc. Are considered. The functions include technical management, application management, operations management and service desk as well as, responsibilities for staff engaging in Service Operation.

List of processes:

    Event management

    Incident management

    Request fulfillment

    Problem management

    Access management


Service desk

The service desk is one of four ITIL functions and is primarily associated with the Service Operation lifecycle stage. Tasks include handling incidents and requests, and providing an interface for other ITSM processes. Features include:

    Single point of contact (SPOC) and not necessarily the first point of contact (FPOC)

    Single point of entry

    Single point of exit

    Easier for customers

    Data integrity

    Streamlined communication channel

Primary purposes of a service desk include:

    Incident control: life-cycle management of all service requests

    Communication: keeping a customer informed of progress and advising on workarounds

The service desk function can have various names, such as:

    Call center: main emphasis on professionally handling large call volumes of telephone-based transactions

    Help desk: manage, co-ordinate and resolve incidents as quickly as possible at primary support level

    Service desk: not only handles incidents, problems and questions but also provides an interface for other activities such as change requests, maintenance contracts, software licenses, service-level management, configuration management, availability management, financial management and IT services continuity management

The three types of structure for consideration:

    Local service desk: to meet local business needs – practical only until multiple locations requiring support services are involved

    Central service desk: for organisations having multiple locations – reduces operational costs[citation needed] and improves usage of available resources

    Virtual service desk: for organisations having multi-country locations – can be situated and accessed from anywhere in the world due to advances[when?] In network performance and telecommunications, reducing operational costs[citation needed] and improving usage of available resources

Application management

ITIL application management  encompasses a set of best practices proposed to improve the overall quality of IT software development and support through the life-cycle of software development projects, with particular attention to gathering and defining requirements that meet business objectives.

Software asset management (SAM) is a primary topic of itilv2 and is closely associated with the ITIL Application Management function. SAM is the practice of integrating people, processes, and technology to allow software licenses and usage to be systematically tracked, evaluated, and managed. The goal of SAM is to reduce IT expenditures, human resource overhead and risks inherent in owning and managing software assets.

SAM practices include:

    Maintaining software license compliance

    Tracking inventory and software asset use

    Maintaining standard policies and procedures surrounding definition, deployment, configuration, use, and retirement of software assets and the definitive software library.

SAM represents the software component of IT asset management. This includes hardware asset management because effective hardware inventory controls are critical to efforts to control software. This means overseeing software and hardware that comprise an organization’s computers and network.

An event may indicate that something is not functioning correctly, leading to an incident being logged. Events may also indicate normal activity, or a need for routine intervention such as changing a tape. Event management depends on monitoring, but it is different. Event management generates and detects notifications, whilst monitoring checks the status of components even when no events are occurring. Events may be detected by a CI sending a message, or by a management tool polling the CI. After an event has been detected it may lead to an Incident, Problem or Change, or it may simply be logged in case the information is needed. Response to an event may be automated or may require manual intervention. If actions are needed then a trigger, such as an SMS message or an incident being automatically logged, can alert support staff.

Incident management

 : Incident management (ITSM)

Incident management aims to restore normal service operation as quickly as possible and minimise the adverse effect on business operations, thus ensuring that the best possible levels of service quality and availability are maintained. ‘Normal service operation’ is defined here as service operation within service-level agreement (SLA) limits.

An incident is defined as:

    2007: An unplanned interruption to an IT service or a reduction in the quality of an IT service. Failure of a configuration item that has not yet impacted service is also an incident. For example, failure of one disk from a mirror set.

    V2: An event which is not part of the standard operation of a service and which causes or may cause disruption to or a reduction in the quality of services and customer productivity.

The objective of incident management is to restore normal operations as quickly as possible with the least possible impact on either the business or the user, at a cost-effective price. The transformation between event-to-incident is the critical junction where Application Performance Management (APM) and ITIL come together to provide tangible value back to the business.[13]

Request fulfillment

Request fulfillment (or request management) focuses on fulfilling Service Requests, which are often minor (standard) changes (e.g., requests to change a password) or requests for information.

Problem management

 : Problem management

Problem management aims to resolve the root causes of incidents and thus to minimise the adverse impact of incidents and problems on business that are caused by errors within the IT infrastructure, and to prevent recurrence of incidents related to these errors. A ‘problem’ is the unknown underlying cause of one or more incidents, and a ‘known error’ is a problem that is successfully diagnosed and for which either a work-around or a permanent resolution has been identified. The CCTA (Central Computer and Telecommunications Agency) defines problems and known errors as follows

    A problem is a condition often identified as a result of multiple incidents that exhibit common symptoms. Problems can also be identified from a single significant incident, indicative of a single error, for which the cause is unknown, but for which the impact is significant.

    A known error is a condition identified by successful diagnosis of the root cause of a problem, and the subsequent development of a work-around.

Problem management differs from incident management. The principal purpose of problem management is to find and resolve the root cause of a problem and thus prevent further incidents; the purpose of incident management is to return the service to normal level as soon as possible, with smallest possible business impact.

The problem-management process is intended to reduce the number and severity of incidents and problems on the business, and report it in documentation to be available for the first-line and second line of the help desk. The proactive process identifies and resolves problems before incidents occur. Such processes include:

    Trend analysis

    Targeting support action

    Providing information to the organisation

The error control process iteratively diagnoses known errors until they are eliminated by the successful implementation of a change under the control of the Change Management process.

The problem control process aims to handle problems in an efficient way. Problem control identifies the root cause of incidents and reports it to the service desk. Other activities are:

    Problem identification and recording

    Problem classification

    Problem investigation and diagnosis

A technique for identifying the root cause of a problem is to use an Ishikawa diagram, also referred to as a cause-and-effect diagram, tree diagram, or fishbone diagram. Alternatively, a formal Root Cause Analysis method such as Apollo Root Cause Analysis can be implemented and used to identify causes and solutions. An effective root cause analysis method and/or tool will provide the most effective/efficient solutions to address problems in the Problem Management process.

Identity management/access and identity management

Identity management (idm) less commonly called Access and Identity Management (AIM) as a process focuses on granting authorised users the right to use a service, while preventing access to non-authorised users. Certain identity management processes executes policies defined in Information Security Management System.

Continual service improvement (CSI)

Continual service improvement, defined in the ITIL continual service improvement volume,[9] aims to align and realign IT services to changing business needs by identifying and implementing improvements to the IT services that support the business processes. It incorporates many of the same concepts articulated in the Deming Cycle of Plan-Do-Check-Act. The perspective of CSI on improvement is the business perspective of service quality, even though CSI aims to improve process effectiveness, efficiency and cost effectiveness of the IT processes through the whole lifecycle. To manage improvement, CSI should clearly define what should be controlled and measured.

CSI needs to be treated just like any other service practice.[citation needed] There needs to be upfront planning, training and awareness, ongoing scheduling, roles created, ownership assigned,and activities identified to be successful. CSI must be planned and scheduled as process with defined activities, inputs, outputs, roles and reporting. Continual Service Improvement and Application Performance Management (APM) are two sides of the same coin. They both focus on improvement with APM tying together service design, service transition, and service operation which in turn helps raise the bar of operational excellence for IT.[14]

Improvement initiatives typically follow a seven-step process:

    Identify the strategy for improvement

    Define what you will measure

    Gather the data

    Process the data

    Analyse the information and data

    Present and use the information

    Implement improvement

Overview of ITIL v2

The eight ITIL version 2 books and their disciplines are:

The IT service management sets

    1. Service Support

    2. Service Delivery

Other operational guidance

    3. ICT infrastructure management

    4. Security management

    5. Application management

    6. Software asset management

To assist with the implementation of ITIL practices a further book was published (Apr 9, 2002) providing guidance on implementation (mainly of Service Management):

    7. Planning to implement service management

And this has more recently (Jan 26, 2006) been supplemented with guidelines for smaller IT units, not included in the original eight publications:

    8. ITIL Small-scale implementation

Service support

The Service Support[15] ITIL discipline focuses on the User of the IT services and is primarily concerned with ensuring that they have access to the appropriate services to support the business functions.

To a business, customers and users are the entry point to the process model. They get involved in service support by:

    Asking for changes

    Needing communication, updates

    Having difficulties, queries

    Real process delivery

The service desk functions as the single contact-point for end-users’ incidents. Its first function is always to document (“create”) an incident. If there is a direct solution, it attempts to resolve the incident at the first level. If the service desk cannot solve the incident then it is passed to a 2nd/3rd level group within the incident management system. Incidents can initiate a chain of processes: incident management, problem management, change management, release management and configuration management. This chain of processes is tracked using the configuration management database (CMDB), – ITIL refers to configuration management system (CMS), which records each process, and creates output documents for traceability (quality management). Note – CMDB/CMS does not have to be a single database. The solution can be Federated.

Service delivery

The service delivery  discipline concentrates on the proactive services the ICT must deliver to provide adequate support to business users. It focuses on the business as the customer of the ICT services (compare with: service support). The discipline consisted of the following processes:

    Service level management

    Capacity management

    IT service continuity management

    Availability management

    Financial management

ICT infrastructure management

Information and Communication Technology (ICT) management[17] processes recommend best practice for requirements analysis, planning, design, deployment and ongoing operations management and technical support of an ICT infrastructure.

The infrastructure management processes describe those processes within ITIL that directly relate to the ICT equipment and software that is involved in providing ICT services to customers.

    ICT design and planning

    ICT deployment

    ICT operations

    ICT technical support

These disciplines are less well understood than those of service management and therefore often some of their content is believed to be covered ‘by implication’ in service management disciplines.

ICT design and planning

ICT design and planning provides a framework and approach for the strategic and technical design and planning of ICT infrastructures. It includes the necessary combination of business (and overall IS) strategy, with technical design and architecture. ICT design and planning drives both the procurement of new ICT solutions through the production of statements of requirement (“SOR”) and invitations to tender (“ITT”) and is responsible for the initiation and management of ICT Programmes for strategic business change. Key outputs from design and planning are:

    ICT strategies, policies and plans

    The ICT overall architecture & management architecture

    Feasibility studies, itts and sors

    Business cases

ICT deployment management

ICT deployment provides a framework for the successful management of design, build, test and roll-out (deploy) projects within an overall ICT programme. It includes many project management disciplines in common with PRINCE2, but has a broader focus to include the necessary integration of release management and both functional and non functional testing.

ICT operations management

ICT operations management provides the day-to-day technical supervision of the ICT infrastructure. Often confused with the role of incident management from service support, operations has a more technical bias and is concerned not solely with incidents reported by users, but with events generated by or recorded by the infrastructure. ICT operations may often work closely alongside incident management and the service desk, which are not-necessarily technical, to provide an ‘operations bridge’. Operations, however should primarily work from documented processes and procedures and should be concerned with a number of specific sub-processes, such as: output management, job scheduling, backup and restore, network monitoring/management, system monitoring/management, database monitoring/management storage monitoring/management. Operations are responsible for the following:

    A stable, secure ICT infrastructure

    A current, up to date operational documentation library (“ODL”)

    A log of all operational events

    Maintenance of operational monitoring and management tools.

    Operational scripts

    Operational procedures

ICT technical support

ICT technical support is the specialist technical function for infrastructure within ICT. Primarily as a support to other processes, both in infrastructure management and service management, technical support provides a number of specialist functions: research and evaluation, market intelligence (particularly for design and planning and capacity management), proof of concept and pilot engineering, specialist technical expertise (particularly to operations and problem management), creation of documentation (perhaps for the operational documentation library or known error database). There are different levels of support under the ITIL structure, these being primary support level, secondary support level and tertiary support level, higher-level administrators being responsible for support at primary level.

The Known Error Database (KEDB) database contains all known error records. This database is created by problem management and used by incident management and problem management, and as part of service knowledge management systems.[18]

Planning to implement service management

The ITIL discipline – planning to implement service management[19] attempts to provide practitioners with a framework for the alignment of business needs and IT provision requirements. The processes and approaches incorporated within the guidelines suggest the development of a continuous service improvement program (CSIP) as the basis for implementing other ITIL disciplines as projects within a controlled program of work. Planning to implement service management focuses mainly on the service management processes, but also applies generically to other ITIL disciplines. Components include:

    Creating vision

    Analysing organisation

    Setting goals

    Implementing IT service management

Small-scale implementation

ITIL Small-scale implementation[20] provides an approach to ITIL framework implementation for smaller IT units or departments. It is primarily an auxiliary work that covers many of the same best practice guidelines as planning to implement service management, service support, and service delivery but provides additional guidance on the combination of roles and responsibilities, and avoiding conflict between ITIL priorities.

Related frameworks

A number of frameworks exist in the field of IT Service Management alongside ITIL.


Microsoft Operations Framework

The Microsoft Operations Framework (MOF) is based on ITIL v2. While ITIL deliberately aims to be platform-agnostic, MOF is designed by Microsoft to provide a common management framework for its products. Microsoft has mapped MOF to ITIL as part of their documentation of the framework.[21]


The British Educational Communications and Technology Agency (BECTA) used ITIL as the basis for their development of Framework for ICT Technical Support (FITS). Their aim was to develop a framework appropriate for British schools, which often have very small IT departments. FITS became independent from BECTA in 2009 and is now maintained and supported by The FITS Foundation. FITS is now used in excess of a thousand schools in the UK, Australia and Norway as the standard for ICT Service Management in the Education sector (Video: What people are saying).

Other frameworks

ITIL is generally equivalent to the scope of the ISO/IEC 20000 standard (previously BS 15000).[22] While it is not possible for an organization to be certified as being ITIL compliant, certification of an organisation is available for ISO20000 [2].

COBIT is an IT governance framework and supporting toolset developed by ISACA. ISACA view ITIL as being complementary to COBIT. They see COBIT as providing a governance and assurance role while ITIL providing guidance for service management.[23]

The enhanced Telecom Operations Map etom published by the telemanagement Forum offers a framework aimed at telecommunications service providers. In a joined effort, TM Forum and itsmf developed an Application Note to etom (GB921) that shows how the two frameworks can be mapped to each other. It addresses how etom process elements and flows can be used to support the processes identified in ITIL.[24][25]

IBM Tivoli Unified Process (ITUP) is aligned with ITIL, but is presented as a complete, integrated process model compatible with IBM’s products.



An itilv2 Foundation Badge.

The certification scheme differs between ITIL v2 and ITIL 2007/2011, and bridge examinations (now retired) allowed owners of v2 certificates to transfer to the new program.[citation needed] ITIL v2 offers three certification levels: Foundation, Practitioner and Manager. These were progressively discontinued in favor of the new scheme introduced along with the publication of the 2007 Edition. ITIL certification levels are now: Foundation, Intermediate, Expert and Master. In addition, the single-process practitioner certifications that were offered by OGC for version 2 have now been replaced and the offering expanded by what are known are complementary certification.

The ITIL certification scheme now offers a modular approach. Each qualification is assigned a credit value; so that upon successful completion of the module, the candidate is rewarded with both a certification and a number of credits. At the lowest level – Foundation – candidates are awarded a certification and two credits. At the Intermediate level, a total of 15 or 16 credits can be earned. These credits may be accumulated in either a “Lifecycle” stream or a “Capability” stream; or combination thereof. Each Lifecycle module and exam is three credits. Each Capability module and corresponding exam is four credits. A candidate wanting to achieve the Expert level will have, among other requirements, to gain the required number of credits  . That is accomplished with two from Foundations, then 15 or 16 from Intermediate, and finally five credits from the “Managing Across the Lifecycle” exam. Together, the total of 22 or 23 earned credits allows a person to request designation as an ITIL Expert.

The complementary certifications also have point values, ranging from 0.5 to 1.5 credits, which can be applied towards ITIL Expert certification. However, only a maximum of six credits from complementary certifications can be applied towards the Expert certification.

The ITIL Certification Management Board (ICMB) manages ITIL certification. The Board includes representatives from interested parties within the community around the world. Members of the Board include (though are not limited to) representatives from the UK Office of Government Commerce (OGC), APM Group (APMG), The Stationery Office (TSO), ITIL Examination Panel, Examination Institutes (eis) and the IT Service Management Forum International (itsmf) as the recognised user group.

Since the early 1990s, EXIN and ISEB had been setting up the ITIL based certification program, developing and providing ITIL exams at three different levels: Foundation, Practitioner and Manager. EXIN[30] and BCS/ISEB[31] (the British Computer Society) had from that time onwards been the only two examination providers in the world to develop formally acknowledged ITIL certifications, provide ITIL exams and accredit ITIL training providers worldwide. These rights were obtained from OGC, the British government institution and owner of the ITIL trademark.[citation needed] OGC signed over the management of the ITIL trademark and the accreditation of examination providers to APM Group in 2006. Now, after signing a contract with EXIN,[30] BCS/ISEB, LOYALIST CERTIFICATION SERVICES [3],PEOPLECERT Group and other certification bodies, APM Group has accredited them as official examination bodies, to offer ITIL exams and accredit ITIL training providers.[citation needed]

On July 20, 2006, the OGC signed a contract with the APM Group to become its commercial partner for ITIL accreditation from January 1, 2007.[32] APMG manage the ITIL Version 3 exams.

APMG maintains a voluntary register of ITIL certified practitioners at their Successful Candidate Register.[33]


Following the passing an APMG/EXIN exam in IT service management (based on ITIL), some people will wear a metal pin on their shirt or jacket. This badge, provided by the ITSMF with basic gold color is set in the form of the ITIL-logo. The ITIL pins consist of a small, diamond-like structure. The meaning and the shape of the diamond is meant to depict coherence in the IT industry (infrastructure as well). The four corners of the pin symbolise service support, service delivery, infrastructure management and IT management.

There are five colors of ITIL pins – each corresponds to the color of the associated core publication:

    ITIL Foundation Badge (Pastel Green). This ITIL lapel pin takes its color from the ITIL Service Strategy book and is awarded on successful completion of the ITIL Foundation exam.

    ITIL Intermediate Capability Badge (Burgundy). There are four ITIL Capability courses. (RCV, OSA, SOA, PPO). You are able to apply for this lapel pin once you have passed each exam. Some examination institutes such as APMG International will send the pins automatically with the candidate’s certificate. This badge shares its color with the ITIL Service Transition book.

    ITIL Intermediate Lifecycle Badge (Teal). For each of the five ITIL Lifecycle courses (SS, SD, ST, SO, CSI), candidates receive this lapel pin after passing the exam. The color for this pin is based on the ITIL Service Operation book.

    ITIL Expert Badge (Lilac). This is currently the highest qualification available with ITIL. The lapel pin is awarded a candidate attains 22 credits through a combination of ITIL training courses. The pin takes its color from the ITIL Continual Service Improvement book.

    ITIL Master Badge (Purple, with the letter M in the middle). Currently in pilot phase this qualification has no training course or exam associated with it. To gain qualification as an ITIL Master, candidates have to have his/her work peer-reviewed by a panel of experts. Once an ITIL Expert has achieved this status, the ITIL Master can wear a lapel pin based on the color of the ITIL Service Design book.

There are three colors of ITIL V2 pins:

    Itilv2 Foundation Badge (green)

    Itilv2 Practitioner Badge (blue)

    Itilv2 Manager Badge (red)

Exam candidates who have successfully passed the examinations for ITIL will receive their appropriate pin from APMG, EXIN or their certification provider regional office or agent.


Organizations and management systems cannot claim certification as “ITIL-compliant”. An organization that has implemented ITIL guidance in IT Service Management (ITSM), may however, be able to achieve compliance with and seek certification under ISO/IEC 20000. Note that there are some significant differences between ISO/IEC20000 and ITIL[34]

    ISO20000 only recognises the management of financial assets, not assets which include “management, organization, process, knowledge, people, information, applications, infrastructure and financial capital”, nor the concept of a “service asset”. So ISO20000 certification does not address the management of ‘assets’ in an ITIL sense.

    ISO20000 does not recognise Configuration Management System (CMS) or Service Knowledge Management System (SKMS), and so does not certify anything beyond Configuration Management Database (CMDB).

    An organization can obtain ISO20000 certification without recognising or implementing the ITIL concept of Known Error, which is usually considered essential to ITIL.


Unbalanced scales.svg

            This article’s Criticism or Controversy section may compromise the article’s neutral point of view of the subject. Please integrate the section’s contents into the article as a whole, or rewrite the material. (February 2013)

ITIL has been criticised on several fronts, including:

    The books are not affordable for non-commercial users

    Implementation and accreditation requires specific training

    Debate over ITIL falling under BSM or ITSM frameworks

    The ITIL details are not aligned with the other frameworks like ITSM

Rob England (also known as “IT Skeptic”) has criticised the protected and proprietary nature of ITIL. He urges the publisher, Cabinet Office, to release ITIL under the Open Government Licence (OGL).

CIO Magazine columnist Dean Meyer has also presented some cautionary views of ITIL,[37] including five pitfalls such as “becoming a slave to outdated definitions” and “Letting ITIL become religion.” As he notes, “…it doesn’t describe the complete range of processes needed to be world class. It’s focused on … Managing ongoing services.”

In a 2004 survey designed by Noel Bruton (author of “How to Manage the IT Helpdesk” and “Managing the IT Services Process”), organisations adopting ITIL were asked to relate their actual experiences in having implemented ITIL. Seventy-seven percent of survey respondents either agreed or strongly agreed that “ITIL does not have all the answers”. ITIL exponents accept this, citing ITIL’s stated intention to be non-prescriptive, expecting organisations to engage ITIL processes with existing process models. Bruton notes that the claim to non-prescriptiveness must be, at best, one of scale rather than absolute intention, for the very description of a certain set of processes is in itself a form of prescription.[38]

While ITIL addresses in depth the various aspects of service management, it does not address enterprise architecture in such depth. Many of the shortcomings in the implementation of ITIL do not necessarily come about because of flaws in the design or implementation of the service management aspects of the business, but rather the wider architectural framework in which the business is situated. Because of its primary focus on service management, ITIL has limited utility in managing poorly designed enterprise architectures, or how to feed back into the design of the enterprise architecture.

Closely related to the architectural criticism, ITIL does not directly address the business applications which run on the IT infrastructure; nor does it facilitate a more collaborative working relationship between development and operations teams. The trend toward a closer working relationship between development and operations is termed: devops. This trend is related to increased application release rates and the adoption of agile software development methodologies. Traditional service management processes have struggled to support increased application release rates – due to lack of automation – and/or highly complex enterprise architecture.

Some researchers group ITIL with lean, Six Sigma and Agile software development operations management.[citation needed] Applying Six Sigma techniques to ITIL brings the engineering approach to ITIL’s framework. Applying Lean techniques promotes continuous improvement of the ITIL’s best practices. However, ITIL itself is not a transformation method, nor does it offer one. Readers are required to find and associate such a method. Some vendors have also included the term Lean when discussing ITIL implementations, for example “Lean-ITIL”. The initial consequences of an ITIL initiative tend to add cost with benefits promised as a future deliverable.[citation needed] ITIL does not provide usable methods “out of the box” to identify and target waste, or document the customer value stream as required by Lean, and measure customer satisfaction.[who?]

ITIL Implementation scenario

You can’t change people. This is a known ideology and maybe a true one. People must change themselves. Perhaps this is why Corporate Executives, IT Directors and others are slow to implement ITIL best practices. Many like the ITIL approach and the projected cost savings but don’t know exactly where to start.
Some feel company culture must change before any other type of change within IT departments will be accepted. Others think everyone must be an ITIL believer before the changes can take place.
So how can a company start with ITIL? In a recent poll done by Benchmark
Learning most companies started with a blend of the ITIL processes Incident, Problem, Change and Release Management; however, the most popular focus was on Change Management. Practitioners like the ability to change things in an orderly way, the increased efficiency and the alignment of IT Services to business requirements and goals. The result is a shift in culture change. As one ITIL practitioner stated, “We led by example but did not make people drink. We made them thirsty.”
For other organizations, simply setting up a Single Point of Contact (SPOC) in a Service Desk, where Incidents are recorded, has resulted in improved customer service. Maybe it wasn’t overnight, but slowly internal customers and end-users started to change their behavior as how they utilized IT changed and as their level of satisfaction rose. The end-user was being ‘fixed.’ As a result people were starting to change, if only in their perception of IT, and so was company culture. Henry David Thoreau said, “Things do not change, we change.” So make a change in your IT processes; start slowly by changing yourself. The overall cultural change you can affect may be great.

The main benefits of ITIL include:
• Alignment with business needs. ITIL becomes an asset to the business when IT can proactively recommend solutions as a response to one or more business
needs. The IT Strategy Group recommended in Service Strategy and the implementation of Service Portfolio Management gives IT the opportunity to understand the business’ current and future needs and develop service offerings that can address them.
• Negotiated achievable service levels.
Business and IT become true partners when they can agree upon realistic service levels that deliver the necessary value at an acceptable cost.
• Predictable, consistent processes. Customer expectations can be set and are easier to meet through the use of predictable processes that are consistently used. Also, good practice processes are foundational and can assist in laying the groundwork to meet
regulatory compliance requirements.
• Efficiency in service delivery. Welldefined processes with clearly documented accountability for each activity as recommended through the use of a RACI matrix can significantly increase the efficiency of processes. In conjunction with the evaluation of
efficiency metrics that indicate the time required to perform each activity, service
delivery tasks can be optimized.
• Measurable, improvable services and processes. The adage that you can’t manage what you can’t measure rings true here. Consistent, repeatable processes can be measured and therefore can be better tuned for accurate delivery and overall effectiveness. For example, presume that a critical success factor for incident management is to reduce the time to restore service. When predictable, consistent processes are used key performance indicators such as Mean Time To Restore Service can be captured to determine whether this KPi is trending in a positive or negative direction so that the appropriate adjustments can be made. Additionally, under ITIL guidelines, services are
designed to be measurable. With the proper metrics and monitoring in place, IT organizations can monitor SLAs and make improvements as necessary.
• A common language – terms are defined.