Building infrastructure around MongoDB, the model

Lately I’ve spent time designing a persistence layer and the tools and implementation to store JSON documents in MongoDB, a NoSQL database. Documents and collections have no schema. In our case, and I’d be surprised if people generally start out more systematically, this meant we didn’t think much about document structure and simply stored things however it suited us at the time. That’s pleasant and works for a while. Trouble starts when you need to process documents automatically, for example to back up documents for a particular entity or check that the documents they reference exist. You need a model describing documents, collections and their relationships, unless you want to code it by hand for every document type.

We described the data in MongoDB with a simple model represented by a JSON document, itself stored in MongoDB. The model serves tools for maintenance operations such as backup and restoration, validation and cleaning up orphaned data. We also want it to drive the persistence layer that stores our documents, though that isn’t implemented yet.

Here’s the structure of the model.

DocumentStorageModel represents the model itself. Its version lets us check the compatibility of associated data. A typical case is a backup I want to restore. I’d like to know that the current data, and thus the runtime, are compatible so I don’t get unpleasant surprises after restoring it. The version also guides operations such as data migration.

DocumentStorageModel contains a list of Collection entries. Each collection has the name it uses in MongoDB and a Documentation entry. That gives a human-readable description of the kind of document it holds, the release that introduced it, and the URI template under which the REST API exposes it. A collection can also have ToolHints, which describes it for tools that create or validate the collection definition directly in MongoDB. For example, temporary data goes into a capped collection, which automatically removes the oldest documents. The hints let us verify that the collection is actually configured that way.

DataDefinition describes the documents and their structure. You can define a Relax NG schema to validate them. Lukáš Křečan wrote more about that in Converting JSON to XML. You can use it both in the persistence layer when changing or writing documents and for verification after a data migration, for example.

Relationships deserve a section of their own. References would be a better name here, to avoid giving the impression that we’re modeling a relational database. We link documents using URI references. We have three types, a reference to itself (SelfDocumentRelation), one to another MongoDB document (InternalDocumentRelation), and one to an entity or document outside our system (ExternalDocumentRelation). Each relationship has two attributes, the path to the field where it appears in the document and a URI template that breaks it into segments. This is useful when we need to determine which project or user a document belongs to. Parsing it with the URI template gives us the particular user or project it refers to.

Sample model

{
       "documentStorageModel":{
          "collections":[
             {
                "collection":{
                   "name":"subscriptions",
                   "affiliation":"PROJECT",
                   "resourceUriTemplate":"/gdc/projects/{project}/users/{user}/subscriptions/{subscription}",
                   "documentation":{
                      "description":"Contains notification subscriptions for given user in given project",
                      "releaseVersion":"R59"
                   },
                   "dataDefinition":{
                      "self":{
                         "selfDocumentRelation":{
                            "path":"subscription.meta.uri",
                            "uriTemplate":"/gdc/projects/{project}/users/{user}/subscriptions/{subscription}"
                         }
                      }
                   }
                }
             }
             ... další kolekce
             ],
             "version":"1.0.0"
        }
    }

Sample document described by the model

{
     "subscription" : {
      "channels" : [
       "/gdc/account/profile/xxx/channelConfigurations/4f4b7e5ce4b0413360a0c4d5"
      ],
      "meta" : {
       "title" : "Test subscription",
       "author" : "/gdc/account/profile/xxx",
       "category" : "subscription",
       "updated" : "2012-02-27 14:00:12",
       "created" : "2012-02-27 14:00:12",
       "uri" : "/gdc/projects/FoodMartDemo/users/xxx/subscriptions/4f4b7e5ce4b0413360a0c4d7"
      },
      "message" : {
       "template" : {
        "expression" : "Hello World"
       }
      },
     }
    }

The model tells us the document should have a field at subscription.meta.uri whose value matches the URI template /gdc/projects/{project}/users/{user}/subscriptions/{subscription}. The document instance shows that it does, with /gdc/projects/FoodMartDemo/users/xxx/subscriptions/4f4b7e5ce4b0413360a0c4d7. A quick look at that URI reference tells us it belongs to the FoodMartDemo project.

Conclusion

I wanted to give you a glimpse of where our work with the MongoDB NoSQL database is going at GoodData. A model that describes the data turned out to be necessary. This isn’t peculiar to NoSQL. We already used a similar approach with a traditional relational database at HP SOA Systinet. If you’ve had similar experiences or use a different approach to describing data and driving the tools that work with it, share it in the comments.