Google Code offered in: English - Español - 日本語 - 한국어 - Português - Pусский - 中文(简体) - 中文(繁體)
Google Cloud Storage API for Google App Engine is an experimental, innovative, and rapidly changing new feature for App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Google Cloud Storage API for Google App Engine. We will inform the community when this feature is no longer experimental.
The Google Cloud Storage API allows your application to store and serve your data as opaque blobs known as "objects". While Cloud Storage offers a RESTful API, this documentation describes how to use Cloud Storage on App Engine.
Google App Engine provides an easier way to read from and write to Google Cloud Storage objects , which allows applications to create and serve data objects. These objects are stored within buckets in Cloud Storage but can be additionally accessed by Google App Engine applications through the Google Cloud Storage API. You can interact with the Google Cloud Storage API using the RESTful interface or through the Google Cloud Storage Java API for Google App Engine applications, which is discussed in this document.
Cloud Storage is useful for storing and serving large files because it does not have a size limit for your objects. Additionally, Cloud Storage offers the use of access control lists (ACLs), the ability to resume upload operations if they're interrupted, use of the POST
policy document, and many other features.
Once you have created an object, it cannot be modified. To modify an existing object, you need to overwrite the object with a new object that contains your desired changes.
For more information, see the Google Cloud Storage pricing.
To use the Cloud Storage API, you must complete the following prerequisites:
The Google Cloud Storage API for Java is available for App Engine SDKs 1.6.0 and later. To run some of the samples for this page, you will also need the Google Plugin For Eclipse. For instructions to update or download the Google Plugin for Eclipse and the App Engine SDK, see the downloads page for Google Plugin for Eclipse.
Activate the Cloud Storage service in the Google APIs Console.
Cloud Storage offers a free trial quota term until December 31, 2011, but requires that you provide billing information. You are not charged unless you exceed the free trial quota.
Cloud Storage is made up of buckets, basic containers that hold your data. Buckets hold all your data in the form of objects, individual pieces of data that you upload to Google Cloud Storage. Before you can use the Google Cloud Storage API, you need to create a bucket(s) where you would like to store your data. The easiest way to do so is using the GSUtil tool or the online Google Storage browser that is accessible through the Google APIs Console.
The samples in this document describe how to create new objects in Cloud Storage. To be able to create new objects, you need to give WRITE
permission for your App Engine application to your Cloud Storage bucket using the following steps:
READ-ONLY
Access for Objects
Alternatively, if you want your Google App Engine application to only be able to read certain objects, you can:
Then, add the service account as a project viewer in the Google APIs Console, giving your App Engine application READ-ONLY
access to your objects. You can also set the predefined public-read ACL, which allows all users, whether or not they are authenticated, access to read your object. You won't need to add your application's service account name as a project viewer, but if you have data you do not want to be publicly accessible, do not set the public-read
ACL on your object.
Note that setting READ
permission only on your buckets does not provide READ
permission to any objects inside those buckets. You must set READ
permission on objects individually.
Tip: There are various ways to set the permissions for your App Engine application. The steps described here are the most straightforward, but you can also use Cloud Storage's ACLs to exercise more explicit or fine-grained control.
Applications can use Cloud Storage to read from and write to large files of any type, or to upload, store, and serve large files from users. Files are called "objects" in Google Cloud Storage once they're uploaded.
You can use the Cloud Storage API from your App Engine application to read and write data to Cloud Storage. In order to use the Cloud Storage Java API, you must include the following import
statements at the beginning of your code:
import com.google.appengine.api.files.AppEngineFile; import com.google.appengine.api.files.FileReadChannel; import com.google.appengine.api.files.FileService; import com.google.appengine.api.files.FileServiceFactory; import com.google.appengine.api.files.FileWriteChannel; import com.google.appengine.api.files.GSFileOptions.GSFileOptionsBuilder;
To create an object:
// Get the file service FileService fileService = FileServiceFactory.getFileService(); /** * Set up properties of your new object * After finalizing objects, they are accessible * through Cloud Storage with the URL: * http://commondatastorage.googleapis.com/my_bucket/my_object */ GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder() .setBucket("my_bucket") .setKey("my_object") .setAcl("public-read"); // Create your object AppEngineFile writableFile = fileService.createNewGSFile(optionsBuilder.build());
In this example, we also pass in the public-read
parameter in the setAcl()
method. This allows everyone to read the object. If you do not set this parameter, Cloud Storage sets this parameter as null and uses the default object ACL for that bucket (by default, this is project-private
).
You can use the following optional methods to set ACLs and HTTP headers on your object:
Method | Description | Usage | Default |
---|---|---|---|
setAcl |
Set a predefined ACL on your object. | setAcl("public-read")
If you do not set this parameter, Cloud Storage sets this parameter as null and uses the default object ACL for that bucket (by default, this is |
None |
setMimeType() |
Set a Content-Type header on your object. | setMimeType("text/html") |
binary/octet-stream |
setCacheControl() |
Set a Cache-Control header on your object. | setCacheControl("no-cache") |
See Cache-Control |
setContentEncoding() |
If your object is compressed, specify the compression method using the Content-Encoding header. | setContentEncoding("gzip") |
None |
setContentDisposition() |
Set the Content-Disposition header for your object. | setContentDisposition("attachment;filename=filename.ext") |
None |
addUserMetadata() |
Create and apply Map of custom headers and values. |
addUserMetadata('header1', 'value1', 'header2', 'value2') These custom headers are served in the following format when accessing objects through http://commondatastorage.googleapis.com: x-goog-meta-custom_header: custom_value |
None |
The following code snippet provides an example using some of these optional methods:
GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder() .setBucket("my_bucket") .setKey("my_object") .setAcl("public-read") .setMimeType("text/html") .setUserMetadata("date-created", "092011", "owner", "Jon"); AppEngineFile writableFile = fileService.createNewGSFile(optionsBuilder.build());
To write to an object, you must open a channel for writing:
// Open a channel for writing boolean lockForWrite = false; // Do you want to exclusively lock this object? FileWriteChannel writeChannel = fileService.openWriteChannel(writableFile, lockForWrite);
Next, you can write to the object using standard Java ways:
// For this example, we write to the object using the PrintWriter PrintWriter out = new PrintWriter(Channels.newWriter(writeChannel, "UTF8")); out.println("The woods are lovely and deep."); out.println("But I have promises too keep."); // Close the object without finalizing. out.close(); /** * You can write to this file again, later, by saving the file path. * It is not possible to read the file until you finalize it. * Once you finalize a file, it is not possible to write to it. */ // Save the file path String path = writableFile.getFullPath(); // Lock the file so no one else can access it at the same time lockForWrite = true; // Write to the unfinalized file again in a separate request writableFile = new AppEngineFile(path); writeChannel = fileService.openWriteChannel(writableFile, lockForWrite); writeChannel.write(ByteBuffer.wrap("And miles to go before I sleep.".getBytes()));
Once you are done writing to the object, you need to finalize the file before you can read from it:
// Finalize the object writeChannel.closeFinally();
Your object is now visible to App Engine with the file name /gs/my_bucket/my_object
. If you set your object to be publicly accessible, your object can be accessed using the URL http://commondatastorage.googleapis.com/my_bucket/my_object
.
Once you finalize an object, you can no longer write to it. If you want to modify your file, you will need to create a new object with the same name to overwrite it.
Before you can read an object, you must finalize the object.
To read an object:
boolean lockForRead = false; String filename = "/gs/my_bucket/my_object"; AppEngine readableFile = new AppEngineFile(filename); FileReadChannel readChannel = fileService.openReadChannel(readableFile, lockForRead); // Read the file in whichever way you'd like BufferedRead reader = new BufferedRead(Channels.newReader(readChannel, "UTF"); String line = reader.readLine(); resp.getWriter().println("READ:" + LINE); readChannel.close();
The maximum size of an object that can be read by an application with one API call is 32 megabytes.
The following is a sample application that demonstrates one use of the Cloud Storage API within an App Engine application. In order to successfully run the application, you must be an owner or have WRITE
access to a bucket.
To set up and run this application:
GuestbookServlet.java
file:
import com.google.appengine.api.files.AppEngineFile; import com.google.appengine.api.files.FileReadChannel; import com.google.appengine.api.files.FileService; import com.google.appengine.api.files.FileServiceFactory; import com.google.appengine.api.files.FileWriteChannel; import com.google.appengine.api.files.GSFileOptions.GSFileOptionsBuilder; import java.io.BufferedReader; import java.io.IOException; import java.io.PrintWriter; import java.nio.ByteBuffer; import java.nio.channels.Channels; import javax.servlet.http.*;
doGet()
method in your GuestbookServlet.java
file with the following code, replacing BUCKETNAME
and FILENAME
with your bucket and file names:
/** * Copyright 2011 Google Inc. All Rights Reserved. * Create, Write, Read, and Finalize Cloud Storage objects. * Access your app at: http://myapp.appspot.com/ **/ @SuppressWarnings("serial") public class GuestbookServlet extends HttpServlet { public static final String BUCKETNAME = "YOUR_BUCKET_NAME"; public static final String FILENAME = "YOUR_FILE_NAME"; @Override public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException { resp.setContentType("text/plain"); resp.getWriter().println("Hello, world from java"); FileService fileService = FileServiceFactory.getFileService(); GSFileOptionsBuilder optionsBuilder = new GSFileOptionsBuilder() .setBucket(BUCKETNAME) .setKey(FILENAME) .setMimeType("text/html") .setAcl("public_read") .addUserMetadata("myfield1", "my field value"); AppEngineFile writableFile = fileService.createNewGSFile(optionsBuilder.build()); // Open a channel to write to it boolean lock = false; FileWriteChannel writeChannel = fileService.openWriteChannel(writableFile, lock); // Different standard Java ways of writing to the channel // are possible. Here we use a PrintWriter: PrintWriter out = new PrintWriter(Channels.newWriter(writeChannel, "UTF8")); out.println("The woods are lovely dark and deep."); out.println("But I have promises to keep."); // Close without finalizing and save the file path for writing later out.close(); String path = writableFile.getFullPath(); // Write more to the file in a separate request: writableFile = new AppEngineFile(path); // Lock the file because we intend to finalize it and // no one else should be able to edit it lock = true; writeChannel = fileService.openWriteChannel(writableFile, lock); // This time we write to the channel using standard Java writeChannel.write(ByteBuffer.wrap ("And miles to go before I sleep.".getBytes())); // Now finalize writeChannel.closeFinally(); resp.getWriter().println("Done writing..."); // At this point, the file is visible in App Engine as: // "/gs/BUCKETNAME/FILENAME" // and to anybody on the Internet through Cloud Storage as: // (http://commondatastorage.googleapis.com/BUCKETNAME/FILENAME) // We can now read the file through the API: String filename = "/gs/" + BUCKETNAME + "/" + FILENAME; AppEngineFile readableFile = new AppEngineFile(filename); FileReadChannel readChannel = fileService.openReadChannel(readableFile, false); // Again, different standard Java ways of reading from the channel. BufferedReader reader = new BufferedReader(Channels.newReader(readChannel, "UTF8")); String line = reader.readLine(); resp.getWriter().println("READ:" + line); // line = "The woods are lovely, dark, and deep." readChannel.close(); } }
From Eclipse:
From commandline:
You can upload your application code and files using a command included in the SDK named appcfg.cmd
(Windows) or appcfg.sh
(Mac OS X, Linux).
AppCfg is a multi-purpose tool for interacting with your app on App Engine. The command takes the name of an action, the path to your application's war/
directory, and other options. To upload the app code and files to App Engine, you use the update action.
To upload the app using Windows:
..\appengine-java-sdk\bin\appcfg.cmd update war
To upload the app using Mac OS X or Linux:
../appengine-java-sdk/bin/appcfg.sh update war
Enter your Google Account username and password at the prompts.
http://application-id.appspot.com
.Cloud Storage is a pay-to-use service that offers a generous free trial quota. If you exceed the free trial quota, you will be charged according to the Cloud Storage price sheet.