Google Code offered in: English - Español - 日本語 - 한국어 - Português - Pусский - 中文(简体) - 中文(繁體)
In App Engine push queues, a task is a unit of work to be performed by the application. Each task is an object of the TaskOptions class. Each Task object contains an application-specific URL with a request handler for the task, and an optional data payload that parameterizes the task.
For example, consider a calendaring application that needs to notify an invitee, via email, that an event has been updated. The data payload for this task consists of the email address and name of the invitee, along with a description of the event. The webhook might live at /app_worker/send_email
and contain a function that adds the relevant strings to an email template and sends the email. The app can create a separate task for each email it needs to send.
You can only use push queues within the App Engine environment; if you need to access App Engine tasks from outside of App Engine, use pull queues.
A Java app sets up queues using a configuration file named queue.xml
, in the WEB-INF/
directory inside the WAR. See Java Task Queue Configuration. Every app has a push queue named default
with some default settings.
To enqueue a task, you get a Queue using the QueueFactory, then call its add()
method. You can get a named queue specified in the queue.xml
file using the getQueue()
method of the factory, or you can get the default queue using getDefaultQueue()
. You can call the Queue's add()
method with a TaskOptions instance (produced by TaskOptions.Builder), or you can call it with no arguments to create a task with the default options for the queue.
The following code adds a task to a queue with options:
import com.google.appengine.api.taskqueue.Queue; import com.google.appengine.api.taskqueue.QueueFactory; import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*; // ... Queue queue = QueueFactory.getDefaultQueue(); queue.add(withUrl("/worker").param("key", key));
Tasks added to this queue will execute by calling the request handler at the URL /worker
with the parameter key
. They will execute at the rate set in the queue.xml
file, or the default rate of 5 tasks per second.
App Engine executes push tasks by calling application-specific URLs with request handlers for those tasks. These URLs must be local to your application root directory and specified as a relative URL. You attach this URL to the task definition using the url parameter of the TaskOptions class.
Programmatically referring to a bundled HTTP request in this fashion is sometimes called a "web hook." You can specify web hooks ahead of time, without waiting for their actual execution. Thus, an application can create many web hooks at once and then hand them off to App Engine. The system then processes them asynchronously (by invoking the HTTP request). This web hook model enables efficient parallel processing.
Note: If a task performs sensitive operations (such as modifying important data), you can secure the worker URL to prevent a malicious external user from calling it directly.
A task must finish executing and send an HTTP response value between 200–299 within 10 minutes of the original request. This deadline is separate from user requests, which have a 60-second deadline. If your task's execution nears the limit, App Engine raises a DeadlineExceededException that you can catch to save your work or log progress before the deadline passes. If the task failed to execute, App Engine retries it based on criteria that you can configure.
Tip: Use App Engine Backends if you need to process tasks requiring more than 10 minutes.
Requests from the Task Queue service contain the following HTTP headers:
X-AppEngine-QueueName
, the name of the queue (possibly default
)X-AppEngine-TaskName
, the name of the task, or a system-generated unique ID if no name was specifiedX-AppEngine-TaskRetryCount
, the number of times this task has been retried; for the first attempt, this value is 0X-AppEngine-FailFast
specifies that a task running on a backend fails immediately instead of waiting in a pending queue.X-AppEngine-TaskETA
, the target execution time of the task, specified in microseconds since January 1st 1970.You set the maximum processing rate for the entire queue when you configure the queue. App Engine uses a token bucket algorithm to execute tasks once they've been delivered to the queue. Each queue has a token bucket, and each bucket holds a certain number of tokens. Your app consumes a token each time it executes a task. If the bucket runs out of tokens, the system pauses until the bucket has more tokens. The rate at which the bucket is refilled is the limiting factor that determines the rate of the queue. See Defining Push Queues and Processing Rates for more details.
To ensure that the taskqueue system does not overwhelm your application, it may throttle the rate at which requests are sent. This throttled rate is known as the enforced rate. The enforced rate may be decreased when your application returns a 503 HTTP response code, or if there are no instances able to execute a request for an extended period of time. You can view the enforced rate on the Task Queue tab of the Administration Console.
The order in which tasks are executed depends on several factors:
If a push task request handler returns an HTTP status code within the range 200–299, App Engine considers the task to have completed successfully. If the task returns a status code outside of this range, App Engine retries the task until it succeeds. The system backs off gradually to avoid flooding your application with too many requests, but schedules retry attempts for failed tasks to recur at a maximum of once per hour.
You can also configure your own scheme for task retries using the retry-parameters element in queue.xml
.
When implementing the code for tasks (as worker URLs within your app), it is important to consider whether the task is idempotent. App Engine's Task Queue API is designed to only invoke a given task once; however, it is possible in exceptional circumstances that a task may execute multiple times (such as in the unlikely case of major system failure). Thus, your code must ensure that there are no harmful side-effects of repeated execution.
Setting up a handler for each distinct task (as described in the previous sections) can be cumbersome, as can serializing and deserializing complex arguments for the task—particularly if you have many diverse but small tasks that you want to run on the queue. The Java SDK includes an interface called DeferredTask. This interface lets you define a task as a single method. This interface uses Java serialization to package a unit of work into a Task Queue. A simple return from that method is considered success. Throwing any exception from that method is considered a failure.
Warning! While the DeferredTask API is a convenient way to handle serialization, you have to carefully control the serialization compatibility of objects passed into payload methods. Careful control is necessary because unprocessed objects remain in the task queue, even after the application code is updated. Tasks based on outdated application code will not deserialize properly when the task is decoded with new revisions of the application.
Push tasks reference their implementation via URL. For example, a task which fetches and parses an RSS feed might use a worker URL called /app_worker/fetch_feed
. You can specify this worker URL or use the default. In general, you can use any URL as the worker for a task, so long as it is within your application; all task worker URLs must be specified as relative URLs:
import com.google.appengine.api.taskqueue.Queue; import com.google.appengine.api.taskqueue.QueueFactory; import com.google.appengine.api.taskqueue.TaskOptions.Method; import com.google.appengine.api.taskqueue.TaskOptions.Builder.*; // ... Queue queue = QueueFactory.getDefaultQueue(); queue.add(withUrl("/path/to/my/worker")); queue.add(withUrl("/path?a=b&c=d").method(Method.GET));
If you do not specify a worker URL, the task uses a default worker URL named after the queue:
/_ah/queue/queue_name
A queue's default URL is used if, and only if, a task does not have a worker URL of its own. If a task does have its own worker URL, then it is only invoked at the worker URL, never another. Once inserted into a queue, its url endpoint cannot be changed.
Warning! If a task does not have a worker URL, then the task is invoked against the queue's default URL, even if there is currently no handler defined for the queue's default URL. This results in a 404 which will be available, along with the exact URL that was tried, in your application's logs. Due to the failure state of this 404, the system saves the task and retries it until it is eventually successful. You can clear (or 'purge') tasks that can't complete successfully using the Administration Console.
You can also target tasks to App Engine Backends. Backends allow you to process tasks beyond the 10-minute deadline for task execution. See Push Queues and Backends for more information.
You can prevent users from accessing URLs of tasks by restricting access to administrator accounts. Task queues can access admin-only URLs. You can read about restricting URLs at Security and Authentication. An example you would use in web.xml
to restrict everything starting with /tasks/
to admin-only is:
<security-constraint> <web-resource-collection> <url-pattern>/tasks/*</url-pattern> </web-resource-collection> <auth-constraint> <role-name>admin</role-name> </auth-constraint> </security-constraint>
Note: While task queues can use URL paths restricted with <role-name>admin</role-name>
, they cannot use URL paths restricted with <role-name>*</role-name>
.
For more on the format of web.xml
, see the documentation on the deployment descriptor.
To test a task web hook, sign in as an administrator and visit the URL of the handler in your browser.
The development server and the production server behave differently:
<rate>
and <bucket-size>
attributes of your queues. As a result, tasks are executed as close to their ETA as possible. Setting a rate of 0 doesn't prevent tasks from being executed automatically.To disable automatic execution of tasks, set the following jvm flag:
--jvm_flag=-Dtask_queue.disable_auto_task_execution=true
You can examine and manipulate tasks from the developer console at:
http://localhost:8080/_ah/admin/taskqueue
.
To execute tasks, select the queue by clicking on its name, select the tasks to execute, and click Run Now. To clear a queue without executing any tasks, click Purge Queue.
Push tasks typically must finish execution within 10 minutes. If you have push tasks that require more time or computing resources to process, you can use App Engine Backends to process these tasks outside of the normal limits of App Engine applications. Backends are addressable. When you create the push task, all you need to do is address it to the URL of the backend you wish to process it. The following code sample demonstrates how to create a push task addressed to an instance 1
of a backend named backend1
:
import com.google.appengine.api.taskqueue.Queue; import com.google.appengine.api.taskqueue.QueueFactory; import static com.google.appengine.api.taskqueue.TaskOptions.Builder.*; import com.google.appengine.api.backends.*; // ... queue.add(withUrl("/path/to/my/worker").param("key", key).header("Host", BackendServiceFactory.getBackendService().getBackendAddress("backend1", 1));
Enqueuing a task in a push queue counts toward the following quotas:
The Task Queue Stored Task Bytes quota is configurable in queue.xml by setting <total-storage-limit>
. This quota counts towards your Stored Data (billable) quota.
Execution of a task counts toward the following quotas:
The act of executing a task consumes bandwidth-related quotas for the request and response data, just as if the request handler were called by a remote client. When the task queue processes a task, the response data is discarded.
Once a task has been executed or deleted, the storage used by that task is reclaimed. The reclaiming of storage quota for tasks happens at regular intervals, and this may not be reflected in the storage quota immediately after the task is deleted.
For more information on quotas, see Quotas, and the "Quota Details" section of the Admin Console.
In addition to quotas, the following limits apply to the use of push queues:
Limit | Amount |
---|---|
task object size | 100KB |
number of active queues (not including the default queue) | 10 for free apps 100 for billed apps |
queue execution rate | 500 task invocations per second per queue |
maximum countdown/ETA for a task | 30 days from the current date and time |
maximum number of tasks that can be added in a batch | 100 tasks |
maximum number of tasks that can be added in a transaction | 5 tasks |