English

Using the Datastore with JDO

Storing data in a scalable web application can be tricky. A user could be interacting with any of dozens of web servers at a given time, and the user's next request could go to a different web server than the one that handled the previous request. All web servers need to be interacting with data that is also spread out across dozens of machines, possibly in different locations around the world.

With Google App Engine, you don't have to worry about any of that. App Engine's infrastructure takes care of all of the distribution, replication and load balancing of data behind a simple API—and you get a powerful query engine and transactions as well.

The App Engine datastore is one of several services provided by App Engine with two APIs: a standard API, and a low-level API. By using the standard APIs, you make it easier to port your application to other hosting environments and other database technologies, if you ever need to. Standard APIs "decouple" your application from the App Engine services. App Engine services also provide low-level APIs that exposes the service capabilities directly. You can use the low-level APIs to implement new adapter interfaces, or just use the APIs directly in your app.

App Engine includes support for two different API standards for the datastore: Java Data Objects (JDO) and Java Persistence API (JPA). These interfaces are provided by DataNucleus Access Platform, an open source implementation of several Java persistence standards, with an adapter for the App Engine datastore.

For the guest book, we'll use the JDO interface to retrieve and post messages left by users.

Setting Up DataNucleus Access Platform

Access Platform needs a configuration file that tells it to use the App Engine datastore as the backend for the JDO implementation. In the final WAR, this file is named jdoconfig.xml and resides in the directory war/WEB-INF/classes/META-INF/.

If you are using Eclipse, this file has been created for you as src/META-INF/jdoconfig.xml. This file is automatically copied into war/WEB-INF/classes/META-INF/ when you build your project.

If you are not using Eclipse, you can create the directory war/WEB-INF/classes/META-INF/ directly, or have your build process create it and copy the configuration file from another location. The Ant build script described in Using Apache Ant copies this file from src/META-INF/.

The jdoconfig.xml file should have the following contents:

<?xml version="1.0" encoding="utf-8"?>
<jdoconfig xmlns="http://java.sun.com/xml/ns/jdo/jdoconfig"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://java.sun.com/xml/ns/jdo/jdoconfig">

    <persistence-manager-factory name="transactions-optional">
        <property name="javax.jdo.PersistenceManagerFactoryClass"
            value="org.datanucleus.store.appengine.jdo.DatastoreJDOPersistenceManagerFactory"/>
        <property name="javax.jdo.option.ConnectionURL" value="appengine"/>
        <property name="javax.jdo.option.NontransactionalRead" value="true"/>
        <property name="javax.jdo.option.NontransactionalWrite" value="true"/>
        <property name="javax.jdo.option.RetainValues" value="true"/>
        <property name="datanucleus.appengine.autoCreateDatastoreTxns" value="true"/>
    </persistence-manager-factory>
</jdoconfig>

JDO Class Enhancement

When you create your JDO classes, you use Java annotations to describe how instances should be stored in the datastore, and how they should be recreated when retrieved from the datastore. Access Platform connects your data classes to the implementation using a post-compilation processing step, which DataNucleus calls "enhancing" the classes.

If you are using Eclipse and the Google Plugin, the plugin performs the JDO class enhancement step automatically as part of the build process.

If you are using the Ant build script described in Using Apache Ant, the build script includes the necessary enhancement step.

For more information on JDO class enhancement, see Using JDO.

POJOs and JDO Annotations

JDO allows you to store Java objects (sometimes called Plain Old Java Objects, or POJOs) in any datastore with a JDO-compliant adapter, such as DataNucleus Access Platform. The App Engine SDK includes an Access Platform plugin for the App Engine datastore. This means you can store instances of classes you define in the App Engine datastore, and retrieve them as objects using the JDO API. You tell JDO how to store and reconstruct instances of your class using Java annotations.

Let's create a Greeting class to represent individual messages posted to the guest book.

Create a new class named Greeting in the package guestbook. (Non-Eclipse users, create the file Greeting.java in the directory src/guestbook/.) Give the source file the following contents:

package guestbook;

import java.util.Date;
import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;
import com.google.appengine.api.users.User;

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Greeting {
    @PrimaryKey
    @Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
    private Long id;

    @Persistent
    private User author;

    @Persistent
    private String content;

    @Persistent
    private Date date;

    public Greeting(User author, String content, Date date) {
        this.author = author;
        this.content = content;
        this.date = date;
    }

    public Long getId() {
        return id;
    }

    public User getAuthor() {
        return author;
    }

    public String getContent() {
        return content;
    }

    public Date getDate() {
        return date;
    }

    public void setAuthor(User author) {
        this.author = author;
    }

    public void setContent(String content) {
        this.content = content;
    }

    public void setDate(Date date) {
        this.date = date;
    }
}

This simple class defines 3 properties for a greeting: author, content and date. These three private fields are annotated with @Persistent to tell DataNucleus to store them as properties of objects in the App Engine datastore.

This class defines getters and setters for the properties. These are used only by the application, and are not needed for JDO.

The class also defines a field named id, a Long annotated as both @Persistent and @PrimaryKey. The App Engine datastore has a notion of entity keys, and can represent the key in several different ways on the object. In this case, the key field is a long integer, and is set automatically to a unique numeric ID when the object is saved.

For more information on JDO annotations, see Defining Data Classes.

The PersistenceManagerFactory

Each request that uses the datastore creates a new instance of the PersistenceManager class. It does so using an instance of the PersistenceManagerFactory class.

A PersistenceManagerFactory instance takes time to initialize. Thankfully, you only need one instance for your application, and this instance can be stored in a static variable to be used by multiple requests and multiple classes. An easy way to do this is to create a singleton wrapper class for the static instance.

Create a new class named PMF in the package guestbook (a file named PMF.java In the directory src/guestbook/), and give it the following contents:

package guestbook;

import javax.jdo.JDOHelper;
import javax.jdo.PersistenceManagerFactory;

public final class PMF {
    private static final PersistenceManagerFactory pmfInstance =
        JDOHelper.getPersistenceManagerFactory("transactions-optional");

    private PMF() {}

    public static PersistenceManagerFactory get() {
        return pmfInstance;
    }
}

Creating and Saving Objects

With DataNucleus and the Greeting class in place, the form processing logic can now store new greetings in the datastore.

Edit src/guestbook/SignGuestbookServlet.java as indicated to resemble the following:

package guestbook;

import java.io.IOException;
import java.util.Date;
import java.util.logging.Logger;
import javax.jdo.PersistenceManager;
import javax.servlet.http.*;
import com.google.appengine.api.users.User;
import com.google.appengine.api.users.UserService;
import com.google.appengine.api.users.UserServiceFactory;

import guestbook.Greeting;
import guestbook.PMF;

public class SignGuestbookServlet extends HttpServlet {
    private static final Logger log = Logger.getLogger(SignGuestbookServlet.class.getName());

    public void doPost(HttpServletRequest req, HttpServletResponse resp)
                throws IOException {
        UserService userService = UserServiceFactory.getUserService();
        User user = userService.getCurrentUser();

        String content = req.getParameter("content");
        Date date = new Date();
        Greeting greeting = new Greeting(user, content, date);

        PersistenceManager pm = PMF.get().getPersistenceManager();
        try {
            pm.makePersistent(greeting);
        } finally {
            pm.close();
        }

        resp.sendRedirect("/guestbook.jsp");
    }
}

This code creates a new Greeting instance by calling the constructor. To save the instance to the datastore, it creates a PersistenceManager using a PersistenceManagerFactory, then passes the instance to the PersistenceManager's makePersistent() method. The annotations and bytecode enhancement take it from there. Once makePersistent() returns, the new object is stored in the datastore.

Queries With JDOQL

The JDO standard defines a mechanism for querying persistent objects called JDOQL. You can use JDOQL to perform queries of entities in the App Engine datastore, and retrieve results as JDO-enhanced objects.

For this example, we will keep things simple by writing the query code directly into guestbook.jsp. For a larger application, you may want to delegate the query logic to another class.

Edit war/guestbook.jsp and add the indicated lines so that it resembles the following:

<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%@ page import="java.util.List" %>
<%@ page import="javax.jdo.PersistenceManager" %>
<%@ page import="com.google.appengine.api.users.User" %>
<%@ page import="com.google.appengine.api.users.UserService" %>
<%@ page import="com.google.appengine.api.users.UserServiceFactory" %>
<%@ page import="guestbook.Greeting" %>
<%@ page import="guestbook.PMF" %>

<html>
  <body>

<%
    UserService userService = UserServiceFactory.getUserService();
    User user = userService.getCurrentUser();
    if (user != null) {
%>
<p>Hello, <%= user.getNickname() %>! (You can
<a href="<%= userService.createLogoutURL(request.getRequestURI()) %>">sign out</a>.)</p>
<%
    } else {
%>
<p>Hello!
<a href="<%= userService.createLoginURL(request.getRequestURI()) %>">Sign in</a>
to include your name with greetings you post.</p>
<%
    }
%>

<%
    PersistenceManager pm = PMF.get().getPersistenceManager();
    String query = "select from " + Greeting.class.getName();
    List<Greeting> greetings = (List<Greeting>) pm.newQuery(query).execute();
    if (greetings.isEmpty()) {
%>
<p>The guestbook has no messages.</p>
<%
    } else {
        for (Greeting g : greetings) {
            if (g.getAuthor() == null) {
%>
<p>An anonymous person wrote:</p>
<%
            } else {
%>
<p><b><%= g.getAuthor().getNickname() %></b> wrote:</p>
<%
            }
%>
<blockquote><%= g.getContent() %></blockquote>
<%
        }
    }
    pm.close();
%>

    <form action="/sign" method="post">
      <div><textarea name="content" rows="3" cols="60"></textarea></div>
      <div><input type="submit" value="Post Greeting" /></div>
    </form>

  </body>
</html>

To prepare a query, you call the newQuery() method of a PersistenceManager instance with the text of the query as a string. The method returns a query object. The query object's execute() method performs the query, then returns a List<> of result objects of the appropriate type. The query string must include the full name of the class to query, including the package name.

Rebuild the project and restart the server. Visit http://localhost:8080/. Enter a greeting and submit it. The greeting appears above the form. Enter another greeting and submit it. Both greetings are displayed. Try signing out and signing in using the links, and try submitting messages while signed in and while not signed in.

Tip: In a real world application, it's a good idea to escape HTML characters when displaying user-submitted content, like the greetings in this app. The JavaServer Pages Standard Tag Library (JSTL) includes routines for doing this. App Engine includes the JSTL (and other JSP-related runtime JARs), so you do not need to include them with your app. Look for the escapeXml function in the tag library http://java.sun.com/jsp/jstl/functions. See Sun's J2EE 1.4 Tutorial for more information.

Introducing JDOQL

Our guestbook currently displays all messages ever posted to the system. It also displays them in the order they were created. When our guestbook has many messages, it might be more useful to display only recent messages, and display the most recent message at the top. We can do this by adjusting the datastore query.

You perform a query with the JDO interface using JDOQL, a SQL-like query language for retrieving data objects. In our JSP page, the following line defines the JDOQL query string:

    String query = "select from " + Greeting.class.getName();

In other words, the JDOQL query string is the following:

select from guestbook.Greeting

This query asks the datastore for every instance of the Greeting class saved so far.

A query can filter the results by including criteria the properties of an object must meet for the object to be a result. For example, to retrieve only the Greeting objects whose author property is alfred@example.com, you would use the following query:

select from guestbook.Greeting where author == 'alfred@example.com'

A query can specify the order in which the results ought to be returned in terms of property values. To retrieve all Greeting objects in the reverse of the order in which they were posted (newest to oldest), you would use the following query:

select from guestbook.Greeting order by date desc

A query can limit the results returned to a range of results. To retrieve just the 5 most recent greetings, you would use order by and range together, as follows:

select from guestbook.Greeting order by date desc range 0,5

Let's do this for the guest book application. In guestbook.jsp, replace the query definition with the following:

    String query = "select from " + Greeting.class.getName() + " order by date desc range 0,5";

Post greetings to the guest book until there are more than 5. Only the most recent 5 are displayed, in reverse chronological order.

For more information about queries and JDOQL, see Queries and Indexes.

Next...

Every web application returns dynamically generated HTML from the application code, via templates or some other mechanism. Most web applications also need to serve static content, such as images, CSS stylesheets, or JavaScript files. For efficiency, App Engine treats static files differently from application source and data files. Let's create a CSS stylesheet for this application, as a static file.

Continue to Using Static Files.