<precompilation-enabled>true</precompilation-enabled>
When choosing a platform to power a new application that will be used by popular media sites, some of the criteria we look for are scalability, ease of deployment, and ease of management. As with the nature of news and media, traffic can go up and down without much warning and getting new content out of the door quickly is the name of the game. App Engine has greatly simplified development and deployment of YouTube Direct, a new tool built on YouTube's open APIs that allows media organizations to request, review, and re-broadcast user-submitted videos by embedding the upload functionality of YouTube directly into their own sites. By using the Google Plugin for Eclipse to "click and deploy" new code changes and App Engine's admin console to manage application data, our launch partners -- most of which had never used App Engine before -- were able to easily deploy their own instances of YouTube Direct.
To date, media organizations like The Washington Post, The San Francisco Chronicle, NPR, The Huffington Post and ABC's Good Morning America are using YouTube Direct powered by App Engine's Java runtime and we hope more organizations take advantage of the platform. Visit YouTube Direct's developer page to learn more about it. The project's source is also available on Google Code.
Here are some of the recent developments from the greater developer community.
Nick Johnson recently added a new module to the App Engine Python SDK which allows you to use the task queue service to execute deferred function calls. This library requires minimal configuration and makes it even easier to use tasks. Using it is as simple as calling deferred.defer with the function and arguments you want to call - for example:
from google.appengine.ext import deferreddeferred.defer(logging.info, "In a deferred task")
For more details, see the article.
The JRuby App Engine project has done a lot of work making spin-up time less painful with the most recent release. They've also had some great contributions from the growing community, such as an Image API that is ImageScience compatible. Ruby programmers can now build applications using familiar tools like Sinatra and DataMapper, while having access to the full set of App Engine APIs for Java. Follow future developments on the jruby-appengine blog and watch for talks by the JRuby App Engine team at upcoming conferences: "Scaling on App Engine with Ruby and Duby" at RubyConf and "JRuby on Google App Engine" at JRubyConf.
Open source efforts to provide alternate hosting environments for App Engine applications continue to expand, with the TyphoonAE project joining the existing AppScale project in providing a platform to run App Engine apps. TyphoonAE aims to provide a complete environment for hosting App Engine apps, and so far includes compatible datastore, memcache, task queue and XMPP implementations.
Juraj recently created and open sourced a library for the Java runtime which "generate[s] appropriate paging queries from a base query, and then using these queries to page through the data." This is intended to simplify iteration through all of the results in a very large result set.
If you love RESTful APIs then it would be worth your while to take a look at AEJ Tools. Their library provides "a server module which provides a Rest style access to your datastore data, and a client module which allows you to run queries remotely using the Groovy interactive console. It can be used to run queries, store new entities or as a data import/export tool."
The datastore related libraries just keep right on coming. Full text search is a highly requested feature for App Engine and those behind the open source appengine-search project have create a Python library which "[c]an defer indexing via Task Queue API. Uses Relation index strategy mentioned in Brett Slatkin's Google I/O talk."
Moving beyond the datastore, the developers at geewax have created an advanced testing framwork called GAE Testbed. It covers everything from isolated and quick unit tests to full scale end to end tests and builds on some of the great testing tools for App Engine which are already avilable.
In other news, the folks at techwalla have migrated their party chat app to App Engine and made the source code available. It uses the new XMPP API to simulate a chat room using your favorite XMPP client. You can read more specifics on their blog.
Check out the open source projects wiki page for more open source App Engine applications and utilities. Since the last posting, nearly 15 new applications have been added, so keep checking in as new projects are added regularly. If you have a project of your own to add, just follow the instructions at the bottom of the same page.
That's brings our community update to a close. We're always interested in hearing about new and interesting projects which use App Engine. If you have something to share, drop us a line.
Congratulations to Audrey Roy -- her "Price It By Phone" application won the Google App Engine + Twilio developer contest that ran from September 21st to October 4th. Audrey will receive $1000 in Google App Engine credit, and a Dell Netbook from Twilio.
Audrey hacked together her contest submission in under 48 hours after going to an art museum and discovering that the $60 box she wanted was only $37 on Amazon.com. She used the Amazon Product Advertising API to look up the best used and new prices for books by ISBN number, using any touch tone phone.
Look up Amazon.com book prices by calling (877) 265-8137. Enter the book's 10-digit ISBN number after the prompt. The voice will tell you Amazon.com's lowest new and used prices for the book. Then, to see the list of books that you've already price-checked, with links to their Amazon detail pages, enter your phone number at http://price-it.appspot.com.
You can find an interview of Audrey on Twilio's blog.
Thank you to everyone who participated in the contest.
Twilio runs weekly developer contests to encourage developers to explore the many use cases, technologies, and industries where voice applications can provide useful solutions. They announce a new category each Monday along with the previous week's winner. This week's category is "Twilio for Salesforce, with Appirio" and the deadline is midnight on November 19th.
Questions? Drop us a line at help@twilio.com
Since the Task Queue API launched in June, we've been thrilled withhow many developers have started using the API.Many developers have asked for more quota so that they can better utilize the API within their apps. We're pleased that we can now address these requests bysignificantly increasing the Task Queue quotas and are happy toannounce the following changes, effective from today:
Stay tuned for more enhancements to the Task Queue system in the coming months.
We're pleased to announce that you can now submit and comment on App Engine news and articles on a new App Engine specific section of the popular social media site, reddit. Our subreddit is for any articles, blog posts, or other content relevant to App Engine, and is actively moderated and maintained by the App Engine team. Registration takes seconds and allows you to submit, vote, and comment on new App Engine content. The top articles, as selected by the App Engine community, are shown in a widget on http://code.google.com/appengine/.
Be sure to check us out on the App Engine reddit - http://reddit.com/r/appengine
Due to two issues introduced in the 1.2.6 release of the Python SDK, we are releasing version 1.2.7 today. This is a bugfix-only release with just a few minor changes (no new features or functionality); all users of App Engine for Python should upgrade to the new version, available on our Downloads page.
The two issues were:
Both are now fixed in 1.2.7.
There is no corresponding release for the Java language version of App Engine (the Java SDK remains at 1.2.6).
The App Engine team is psyched to present version 1.2.6 of our SDK for both runtimes, Python and Java. This releases contains plenty of improvements and bugfixes, along with some exciting new features:
Incoming Email - Your App Engine app has been able to send email for some time ... but now, with 1.2.6, your app can also receive email. After enabling mail as an inbound service (just like XMPP), users can email your application at whatever@yourappid.appspotmail.com. Inbound messages are converted to HTTP requests (again, just like XMPP) which you can receive via webhook handler. Docs for Python, Java.
Delete an App - Our developers have been quite vocal with their desire to delete an App Engine application once it is no longer used (It's one of the top 5 most requested features on our issue tracker). Well, this feature is now available via the Admin Console! Just visit the Application Settings page for more information. Please be careful when deleting an app - the appid can never be reused after deletion. Learn more.
Datastore Stats - You can now see more detailed statistics about how your application data is stored in the Admin Console. This information can also be accessed programmatically. Docs for Python, Java.
More details can be found in the Release Notes. To download the new SDK and get coding, please visit the Downloads page.
We hope you've been enjoying the new XMPP API we shipped in App Engine release 1.2.5. We're always impressed with the cool scenarios our developers create!
A couple of Google engineers found the time to create cool App Engine demos using the XMPP API and we'd like to share them with you.
CrowdGuru - this app, courtesy of App Engine Developer Programs Engineer Nick Johnson, is a fun trivia game which crowdsources answers. Add crowdguru@appspot.com to your IM list and ask it a question. You can read up on full details of the design and implementation here.
Multi-Chat - David Symonds wasn't satisfied with simple person-to-person IM, so he built an IRC-like chat room system! Add multi-chat@appspot.com to your IM list and message it "/help" to find out how you can join channels and start chatting it up. Full source is available here, and a basic web interface is here.
Of course, please note that these apps are solely intended to demonstrate the power of the XMPP API - they are not official products in any form, nor are they supported. In particular, don't send any private or personally identifying information when chatting with these bots.
But definitely try them out and let us know what cool app experiences you are building with App Engine!
This week, Google App Engine and Twilio challenge you to build voice or voice-enhanced applications hosted on App Engine that use Twilio's voice application API to make and receive phone calls. Submit your app by October 4th to win a Dell Netbook from Twilio and $1000 of App Engine hosting credit from Google.
Twilio hosts a telephony in the cloud web service, allowing web developers to build scalable, reliable voice communications applications on web hosting platforms like Google App Engine. Put Twilio and App Engine together and you can build voice applications with features, scalability, and reliability that companies used to spend millions for.
Voice apps can be as sophisticated as an airline flight status hotline or a company PBX, or as simple as getting the weather by phone. For inspiration, check out what previous contest winners have built:
Where to Submit: http://contests.twilio.com/submit-your-twilio-project.htmlDeadline: October 4th at MIDNIGHT (Pacific Time)Prize: 1 Dell Netbook from Twilio and $1000 in Google App Engine hosting credit
Guest Post by Danielle Morrill, Director of Marketing, Twilio
It's September and the fall conference calendar is starting to fill up. Members of the App Engine team will present at these conferences: join us if you are in the area, and feel free to tweet @app_engine if you want us to participate in local developer community events around these dates!
We look forward to meeting you during these trips, if you can make it.
Posted by Patrick Chanezon, App Engine Team
I work for Norex, a web development company in Halifax, Nova Scotia. As part of our sponsorship of the 2009 ICF Canoe Sprint World Championships in Halifax (Canoe '09), we developed an application to deliver real time race results to standard and mobile web browsers. Thanks to the ability to rapidly develop and deploy a scalable application on Google App Engine, and to do so live during the event, what began as a small experiment became a huge success for Norex.
Our goal was to import instant results (directly from the timing system FinishLynx), upcoming race information, athlete bios, teams, and past race results from a system used by the race organizers, and to reformat that data for distribution to web browsers. We chose to develop in Google App Engine (with Django and appengine patch) and iUI (iPhone User Interface) frameworks. In fact, this was our first time deploying an app using any of these technologies! While we had a few weeks to build the prototype, we also had to deal with the added challenges of a varying data format and special cases that could occur during live races. As with many software projects, these specifications were not provided on a regular or consistent basis from the race organizers, so we had to anticipate the unknowns and work to adjust to the conditions at the time.
To give you a taste for just how agile we were able to be on App Engine, here are some events during the race which we were able to cope with:
We found that new deployments were possible (and quick) because we could stage versions of the application to the App Engine servers, and test these staged versions on live data. Switching from the live version to the staged version and back again takes seconds, thus we could paddle as confidently and rapidly as the racers.
The International Canoe Federation was so impressed by the dependability and versatility of the solution, that they elected to replace their usual channels with our application to serve up the official results for news organizations and to Sport Federation sites all over the world, and to be used internally for calculating medal counts.
All in all, we saw over 1,000,000 page views from 93 countries around the world, and experienced incredible stability and scalability from Google App Engine even when we were spiking 350 requests per second during the finals. It was a big win for Norex, and App Engine has proven itself to be a serious contender for developing scalable web applications. Thanks so much to the Google App Engine team for providing such an outstanding product!
Link to: ZAP Results Application
Guest Post by Adam Thurlow, Web Application Developer, Norex.ca
At Google, we've learned through experience to treat everything with healthy skepticism. We expect that servers, racks, shared GFS cells, and even entire datacenters will occasionally go down, sometimes with little or no warning. This has led us to try as hard as possible to design our products to run on multiple servers, multiple cells, and even multiple datacenters simultaneously, so that they keep running even if any one (or more) redundant underlying parts go down. We call this multihoming. It's a term that usually applies narrowly, to networking alone, but we use it much more broadly in our internal language.
Multihoming is straightforward for read-only products like web search, but it's more difficult for products that allow users to read and write data in real time, like GMail, Google Calendar, and App Engine. I've personally spent a while thinking about how multihoming applies to the App Engine datastore. I even gave a talk about it at this year's Google I/O.
While I've got you captive, I'll describe how multihoming currently works in App Engine, and how we're going to improve it with a release next week. I'll wrap things up with more detail about App Engine's maintenance schedule.
When we launched App Engine, the datastore served each application's data out of one datacenter at a time. Data was replicated to other datacenters in the background, using Bigtable's built-in replication facility. For the most part, this was a big win. It gave us mature, robust, real time replication for all datastore data and metadata.
For example, if the datastore was serving data for some apps from datacenter A, and we needed to switch to serving their data from datacenter B, we simply flipped the datastore to read only mode, waited for Bigtable replication to flush any remaining writes from A to B, then flipped the switch back and started serving in read/write mode from B. This generally works well, but it depends on the Bigtable cells in both A and B to be healthy. Of course, we wouldn't want to move to B if it was unhealthy, but we definitely would if B was healthy but A wasn't.
Google continuously monitors the overall health of App Engine's underlying services, like GFS and Bigtable, in all of our datacenters. However, unexpected problems can crop up from time to time. When that happens, having backup options available is crucial.
You may remember the unplanned outage we had a few months ago. We published a detailed postmortem; in a nutshell, the shared GFS cell we use went down hard, which took us down as well, and it took a while to get the GFS cell back up. The GFS cell is just one example of the extent to which we use shared infrastructure at Google. It's one of our greatest strengths, in my opinion, but it has its drawbacks. One of the most noticeable drawback is loss of isolation. When a piece of shared infrastructure has problems or goes down, it affects everything that uses it.
In the example above, if the Bigtable cell in A is unhealthy, we're in trouble. Bigtable replication is fast, but it runs in the background, so it's usually at least a little behind, which is why we wait for that final flush before switching to B. If A is unhealthy, some of its data may be unavailable for extended periods of time. We can't get to it, so we can't flush it, we can't switch to B, and we're stuck in A until its Bigtable cell recovers enough to let us finish the flush. In extreme cases like this, we might not know how soon the data in A will become available. Rather than waiting indefinitely for A to recover, we'd like to have the option to cut our losses and serve out of B instead of A, even if it means a small, bounded amount of disruption to application data. Following our example, that extreme recovery scenario would go something like this:
We give up on flushing the most recent writes in A that haven't replicated to B, and switch to serving the data that is in B. Thankfully, there isn't much data in A that hasn't replicated to B, because replication is usually quite fast. It depends on the nature of the failure, but the window of unreplicated data usually only includes a small fraction of apps, and is often as small as a few thousand recent puts, deletes, and transaction commits, across all affected apps.
Naturally, when A comes back online, we can recover that unreplicated data, but if we've already started serving from B, we can't automatically copy it over from A, since there may have been conflicting writes in B to the same entities. If your app had unreplicated writes, we can at least provide you with a full dump of those writes from A, so that your data isn't lost forever. We can also provide you with tools to relatively easily apply those unreplicated writes to your current datastore serving out of B.
Unfortunately, Bigtable replication on its own isn't quite enough for us to implement the extreme recovery scenario above. We use Bigtable single-row transactions, which let us do read/modify/write operations on multiple columns in a row, to make our datastore writes transactional and consistent. Unfortunately, Bigtable replication operates at the column value level, not the row level. This means that after a Bigtable transaction in A that updates two columns, one of the new column values could be replicated to B but not the other.
If this happened, and we switched to B without flushing the other column value, the datastore would be internally inconsistent and difficult to recover to a consistent state without the data in A. In our July 2nd outage, it was partly this expectation of internal inconsistency that prevented us from switching to datacenter B when A became unhealthy.
Thankfully, there's a solution to our consistency problem: Megastore replication. Megastore is an internal library on top of Bigtable that supports declarative schemas, multi-row transactions, secondary indices, and recently, consistent replication across datacenters. The App Engine datastore uses Megastore liberally. We don't need all of its features - declarative schemas, for example - but we've been following the consistent replication feature closely during its development.
Megastore replication is similar to Bigtable replication in that it replicates data across multiple datacenters, but it replicates at the level of entire entity group transactions, not individual Bigtable column values. Furthermore, transactions on a given entity group are always replicated in order. This means that if Bigtable in datacenter A becomes unhealthy, and we must take the extreme option to switch to B before all of the data in A has flushed, B will be consistent and usable. Some writes may be stuck in A and unavailable in B, but B will always be a consistent recent snapshot of the data in A. Some scattered entity groups may be stale, ie they may not reflect the most recent updates, but we'd at least be able to start serving from B immediately, as opposed waiting for A to recover.
Megastore replication was originally intended to replicate across multiple datacenters synchronously and atomically, using Paxos. Unfortunately, as I described in my Google I/O talk, the latency of Paxos across datacenters is simply too high for a low-level, developer facing storage system like the App Engine datastore.
Due to that, we've been working with the Megastore team on an alternative: asynchronous, background replication similar to Bigtable's. This system maintains the write latency our developers expect, since it doesn't replicate synchronously (with Paxos or otherwise), but it's still consistent and fast enough that we can switch datacenters at a moment's notice with a minimum of unreplicated data.
We've had a fully functional version of asynchronous Megastore replication for a while. We've been testing it heavily, working out the kinks, and stressing it to make sure it's robust as possible. We've also been using it in our internal version of App Engine for a couple months. I'm excited to announce that we'll be migrating the public App Engine datastore to use it in a couple weeks, on September 22nd.
This migration does require some datastore downtime. First, we'll switch the datastore to read only mode for a short period, probably around 20-30 minutes, while we do our normal data replication flush, and roll forward any transactions that have been committed but not fully applied. Then, since Megastore replication uses a new transaction log format, we need to take the entire datastore down while we drop and recreate our transaction log columns in Bigtable. We expect this to only take a few minutes. After that, we'll be back up and running on Megastore replication!
As described, Megastore replication will make App Engine much more resilient to hiccoughs and outages in individual datacenters and significantly reduce the likelihood of extended outages. It also opens the door to two new options which will give developers more control over how their data is read and written. First, we're exploring allowing reads from the non-primary datastore if the primary datastore is taking too long to respond, which could decrease the likelihood of timeouts on read operations. Second, we're exploring full Paxos for write operations on an opt-in basis, guaranteeing data is always synchronously replicated across datacenters, which would increase availability at the cost of additional write latency.
Both of these features are speculative right now, but we're looking forward to allowing developers to make the decisions that fit their applications best!
Finally, a word about our maintenance schedule. App Engine's scheduled maintenance periods usually correspond to shifts in primary application serving between datacenters. Our maintenance periods usually last for about an hour, during which application serving is continuous, but access to the Datastore and memcache may be read-only or completely unavailable.
We've recently developed better visibility into when we expect to shift datacenters. This information isn't perfect, but we've heard from many developers that they'd like more advance notice from App Engine about when these maintenance periods will occur. Therefore, we're happy to announce below the preliminary maintenance schedule for the rest of 2009.
We don't expect this information to change, but if it does, we'll notify you (via the App Engine Downtime Notify Google Group) as soon as possible. The App Engine team members are personally dedicated to keeping your applications serving without interruption, and we realize that weekday maintenance periods aren't ideal for many. However, we've selected the day of the week and time of day for maintenance to balance disruption to App Engine developers with availability of the full engineering teams of the services App Engine relies upon, like GFS and Bigtable. In the coming months, we expect features like Megastore replication to help reduce the length of our maintenance periods.
Posted by Ryan Barrett, App Engine Team
As recently announced on the Google App Engine Blog, the 1.2.5 SDK for Python now includes a GUI for creating, running, and deploying App Engine applications when developing on Windows. We call this the Google App Engine Launcher.
About a year ago, a few of us recognized a need for a client tool to help with App Engine development. In our 20% time, a we wrote a launcher for the Mac. Of course, not all App Engine developers have Macs, so more work was needed. Thus, a new crew of 20%ers set off to write a launcher for our App Engine developers on Windows. Although Google is spread out across many offices around the world, it is surprisingly easy to connect with passionate engineers. For example, this new launcher for Windows has contributions from Dave Symonds in Australia, Mark Dalrymple on the east coast, and more engineers here in Mountain View.
The Windows launcher is written in Python and uses wxPython for its GUI. This means (with a little care) the launcher should work on Linux, and we'd like Linux developers to have the option of using it. Although we ship a binary of the Launcher for Windows (thanks to py2exe), shipping binaries for Linux is a bit more challenging. Fortunately, Google has a well-traveled path for solving this problem. For example, Google O3D provides binaries for Windows/Mac; it also provides source code and instructions for building on Linux. Thus inspired, we've open sourced the Windows launcher so that developers can use it on other platforms.
The goal of the launcher is to help make App Engine development quick and easy. There may be other tasks you'd like to integrate (e.g. run tests, re-encode images before deploying, etc) and with the launcher now open sourced, you can add them! We look forward to seeing contributions from the community.
We have also started the process of open sourcing the Mac version of the launcher. The source code is now available; however, it references some missing Google libraries, so it won't yet compile in its current state. Fortunately, those libraries have also been open sourced, so it will be possible to get things up and running using entirely open source code. I'll be using more of my 20% time to clean up the Mac launcher project in the coming weeks.
We hope the launcher will improve the workflow for App Engine developers. We also hope the source code will enable developers to adapt it to their needs, just as we do on Chrome, my main project. Finally, I am proud to continue a tradition of openness which began with my very first project at Google.
-- John Grabowski, Software Engineer
Let us know how the launcher works for you.
Open Source Code for the App Engine Launcher: for Windows and Linux, and for Mac OS X.
Today we are releasing version 1.2.5 of the App Engine SDK for both Python and Java, our first simultaneous release across both runtimes. We're excited about the great new functionality in this release ... including XMPP!
XMPP (or Jabber as it is sometimes known) is an open standard for communicating in real-time (instant messaging). One of the most popular API requests in the App Engine issue tracker has been support for XMPP, so today we are excited to mark that issue closed with the release of our new XMPP API for both Python and Java SDKs!
Like the other APIs that App Engine provides for developers, XMPP is built on the same powerful infrastructure that serves other Google products. In this case, we take advantage of the servers that run Google Talk. This new API allows your app to exchange messages with users on any XMPP-based network, including (but not limited to!) Google Talk. If you're currently participating in the Google Wave developer preview, you can also use the XMPP API to build bots that interact with your waves.
We've tried to make the XMPP API as simple as possible to incorporate into your existing Python or Java applications. We use the same webhook pattern that Cron and Task Queue already use: you send outgoing messages with an API call; you receive incoming messages as an HTTP POST. You can read more about the features the XMPP API in our documentation (Python, Java).
We're very proud of our first XMPP release, but there's still more work to do. In the future we hope to provide even more functionality to apps, such as user status (presence) and info on new subscriptions. If you have particular requests or feedback, please let us know.
Python developers have been processing tasks offline using App Engine Task Queues since mid-June, but until now the feature was not available in the App Engine for Java SDK. The 1.2.5 SDK now includes support for creating Tasks and Queues in our Java runtime.
If you're familiar with the Python Task Queue API, the Java version will look very familiar. We use the same webhooks pattern as with Cron (and now XMPP). The API provides a simple pattern for creating tasks, assigning them a payload and a worker, and inserting them into queues for scheduling and processing. There's lots of potential with the Task Queue API, so make sure to check out the Java Task Queue Documentation for more details.
With the 1.2.5 release, we are increasing the daily quota for Task Queue insertions to 100K for billing-enabled apps. Ultimately, we will raise the quota for both free and billing-enabled apps, but we hope this intermediate step opens up new scenarios for our developers using Task Queues.
Last but not least, we're very excited that 1.2.5 for Python now includes a Windows-based version of a useful tool that Mac OS X users have been enjoying for sometime: The Google App Engine Launcher!
This tool simplifies the process of creating new Python projects, testing them locally, and uploading them to the App Engine servers. In addition, we're releasing the source code for both Mac and Windows App Engine Launchers as open source projects. Watch this space for more details on where you can find the source, and how Linux developers can use the Launcher as well.
1.2.5 also includes the usual set of bug fixes, tweaks, and API polish. For a more detailed look at all the things that have changed this release, take a look at our release notes and, as always, let us know what you think!
When we released version 1.2.4 of the SDK earlier this month, a couple of new features were released that didn't quite make it into the release notes. We think they're really cool features, so we wanted to take the time to highlight them.
The first feature is that when logged into your app as an administrator, App Engine will include a couple of extra headers in all the HTTP responses it sends you. As an example, here's what I see in Firefox's Live HTTP headers plugin when I load my blog:X-AppEngine-Resource-Usage: ms=293 cpu_ms=500 api_cpu_ms=236X-AppEngine-Estimated-CPM-US-Dollars: $0.012320
X-AppEngine-Resource-Usage: ms=293 cpu_ms=500 api_cpu_ms=236X-AppEngine-Estimated-CPM-US-Dollars: $0.012320
The first header tells me that it took 293 milliseconds to generate the page, and 500 CPU milliseconds were consumed, of which 236 milliseconds were spent doing API calls - such as accessing memcache or the datastore. The second header tells me that App Engine estimates that serving 1000 requests like this one would cost about $0.01 if I was above my free quota - not bad!
You can view these headers using plugins such as Firefox's Live HTTP Headers or Firebug. Note that only logged in administrators see these figures - ordinary users, and users who aren't logged in, won't see them at all.
The second new feature is that we've enabled 'wildcard' domains for App Engine apps serving off appspot.com. What this means is that you can now create multiple subdomains for your App Engine app, and have them all served by the same application. Thus, your users can access 'myapp.appspot.com', or 'developer.myapp.appspot.com', or 'news.myapp.appspot.com', with your app deciding how to handle each request. No setup is required to use the wildcard domains - simply configure your app to serve up requests to these subdomains however you wish. You can detect which domain a user requested by looking at the 'Host' header in the incoming request - for example, in Python's webapp framework you can access this with self.request.headers['Host'].
Every month or so, we compile a list of interesting things in the community related to App Engine. Here are some recent projects and resources that you might find interesting:
Ubisoft has released a fun game for Facebook called TickTock which challenges you to guess which of your friends have posted particular status updates. From the application page "TickTock is a Facebook trivia game that tests your knowledge of your friends. Use that knowledge to create playful devices to challenge others. Unlock items and create bigger challenges. Post them on your people's Facebook Walls, and see who knows their friends best!"
TickTock uses App Engine to serve the game content and store game information. We're excited to see this as a great example of a growing number of games which are using App Engine behind the scenes. Sign on to Facebook and give it a try.
Written by a group of engineers at VendAsta, asynctools is a rather nifty toolkit that allows you to execute datastore queries in parallel in the Python runtime. The interface is slightly more involved than using standard queries, but the ability to execute multiple queries in parallel can substantially reduce the render time for a page where you need to execute multiple independent queries.
Jason Collins has written a detailed article about how and why they wrote asynctools, which can be found here.
DryDrop is a new tool that lets you host your static site on Google App Engine and update it by pushing to GitHub. Thanks to GitHub post-receive hooks your App Engine site can be updated automatically when you push new content.
We have a new screencast up in the Developer's Channel on YouTube, this one covers Installing Google App Engine applications into Google Apps.
For more open source projects built on top of Google App Engine, see the Open Source Projects wiki page, and feel free to add your own open source projects to that page if you want to see them featured in an upcoming community update post.
Posted by Jeff Scudder, Nick Johnson, and Jason Cooper, App Engine Team
Java is a trademark or registered trademark of Sun Microsystems, Inc. in the United States and other countries.
We're psyched to release version 1.2.4 of the App Engine SDK for Python. Some highlights of what you'll find in this release:
For full details, please see the SdkReleaseNotes wiki page.
Downloads for Windows, Mac, and Linux are available on the Downloads page. This SDK update was for the Python runtime, so please post your feedback in the Python runtime discussion group.
We'd also like to draw your attention to two new articles from App Engine team members:
The App Engine datastore API comes with a wide range of Property classes you can use to represent properties on your datastore models. Occasionally, though, you'd like to do something that the designers didn't think of. Fortunately, it's really easy to extend the Datastore API with custom Property classes. In this blog post, we'll demonstrate how to write your own Property class to store Python's decimal data type.
The Property interface is documented here, but there are two essential methods our DecimalProperty must override: get_value_for_datastore, and make_value_from_datastore. We also need to declare one field, data_type, that specifies the data type our property class will contain. Let's start with a straightforward implementation:
class DecimalProperty(db.Property): data_type = decimal.Decimal def get_value_for_datastore(self, model_instance): return str(super(DecimalProperty, self).get_value_for_datastore(model_instance)) def make_value_from_datastore(self, value): return decimal.Decimal(value)
Note that in the case of get_value_for_datastore, we used super to call the parent class implementation, which handles the details of actually storing and retrieving the data stored in the model. We then simply convert the value to a string for storage in the datastore. In make_value_from_datastore, we reconstruct a Decimal object from the stored string.
That's all that's required for a basic property class! There is something missing from our example, though: We don't perform any validation to make sure that the user actually passed in a Decimal object. To do that, we override the validate method:
def validate(self, value): value = super(DecimalProperty, self).validate(value) if value is None or isinstance(value, decimal.Decimal): return value elif isinstance(value, basestring): return decimal.Decimal(value) raise db.BadValueError("Property %s must be a Decimal or string." % self.name)
Again we start by calling the parent class's implementation, which performs some basic validity checks. Then we check if the value is a Decimal - in which case we return it - or a string, in which case we convert it to a decimal and return it. If the value is invalid, we raise a BadValueError.
Using our property class is as simple as using any other one:
class MyModel(db.Model): a_decimal = DecimalProperty()model = MyModel()model.a_decimal = decimal.Decimal("123.45")model.put()
For a more in depth treatment of writing your own property class, see the article by Rafe Kaplan, Extending Model Properties.
The latest release of Python SDK 1.2.3, which introduced the Task Queue API and integrated support for Django 1.0, may have received a lot of attention, but there have been a number of other notable launches and events since our last update. Several of these are summarized below.
On June 24th, Google Apps and Virgin America invited people from around the world to participate in a one-day online scavenger hunt called Day in the Cloud. Competitors were each given one hour to solve various puzzles and find answers to a host of trivia questions, and prizes were awarded to the top five scorers. The site was hosted on App Engine's scalable infrastructure. For more information on the challenge, including photos and videos of participants competing mid-flight, check out the official Day in the Cloud Lounge.
App Engine enables you to deploy and run your Python- and Java-based web applications on Google's highly scalable infrastructure. There are a number of ways to optimize the performance of an application running on App Engine, some obvious and others more subtle. With this in mind, we recently released a series of articles promoting tips and tricks that you can use to make best use of resources like memcache and prevent potential issues such as datastore contention.
We also updated the articles listing since we recognized that the growing number of articles added since April 2008 (over 35 by last count) made it difficult to find content on a specific topic. The new listing allows you to filter the list of articles based on a particular tag or label, so you can easily find all articles related to the datastore, for example.
As always, we are interested in your comments, thoughts and suggestions for new articles, so please feel free to share.
Google Earth API support engineer Roman Nurrik recently open sourced his GeoModel project. GeoModel uses geohash-like objects called 'geocells' to provide a generalized solution for indexing and querying geospatial data in App Engine. GeoModel is optimized for the basic real estate finder/store locator use case, but can be adapted for use with large datasets.
Using GeoModel, developers can instantly geo-contextualize datastore models by simply inherting from the GeoModel class. Currently, entities can be associated with a single geographic point and subsequently indexed and filtered by either conformance to a bounding box or by proximity (nearest-n) to a search center point.
Posted by Jason Cooper, App Engine Team
With release 1.2.3 of the Python SDK, we are psyched to present an exciting new feature - the Task Queue API. You can now perform offline processing on App Engine by scheduling bundles of work (tasks) for automatic execution in the background. You don't need to worry about managing threads or polling - just write the task processing code, queue up some input data, and App Engine handles the rest. If desired, you can even organize and control task execution by defining custom queues. A quick example:
# for each user, add a task to send a custom email message for u in users: taskqueue.add(url='/work/sendmail', params=dict(to=u.email, subject='Hello ' + u.name, body='this is a message!')) return # finished now, emails will be sent offline when tasks execute ... # task handler at /work/sendmail, automatically called for each task created above class MailWorker(webapp.RequestHandler): def post(self): mail.send_mail( 'from_me@example.com', self.request.get('to'), self.request.get('subject'), self.request.get('body'))
We're eager to help you learn and experiment with Task Queues. The team recently presented the feature at Google I/O and the video is now available (slides are here). We've also prepared a set of demos to help you get started. And of course, don't miss the feature documentation. The Task Queue API is Python-only for now; we'll have a Java language version available soon.
Please note that the Task Queue API is currently a Labs release - we want to get your feedback on its usability and functionality before finalizing the API. You'll notice that its Python import path currently includes the 'labs' module (google.appengine.api.labs.taskqueue). Before the feature is promoted out of Labs, we may need to:
Once we're ready to promote the feature out of Labs, we'll give weeks of notice and provide a transition path for our developers.
Last but not least, the 1.2.3 release is full of other new stuff as well! Stay tuned to the blog for more updates or check the release notes for exciting info on:
Visit the Downloads page to get SDK 1.2.3 now!
The Task Queue API is the first milestone of our plan to deliver rich support for offline processing. There's more to come, but we hope the simplicity and power of this first release opens a new range of possibilities for our developers. Try it out and let us know! We'll be watching the Group for your input.
Since App Engine launched, our goal has been to offer substantial free hosting resources to as many developers as possible. As previously announced, we are changing our free resource quota levels, effective on June 22nd. Our target level of free support has been 5 million page views per month for a reasonably efficient web application.
When we launched App Engine, we were intentionally generous in our free quotas, both because we didn't know resources usage of a typical request, and because we didn't offer a billing feature to allow developers to buy more resources for a higher-traffic app. Since our billing feature launched in February, developers with high-traffic applications can purchase additional resources far beyond our original fixed free quotas. Having been live for more than a year, we now have good empirical data on the average resource consumption per request, so we're able to set our quotas to more accurately support our 5 million page views target.
This change in the free quotas offered to every application is intended to allow us to continue to offer substantial free application hosting to any interested developer. We have grown a lot in the last year, with over 80,000 applications created, and with these changes to our free quotas, more than 90% of these applications will continue to serve completely free. To empirically determine reasonable levels for our quotas, we measured resource usage for all applications running on App Engine over a recent 7-day period. For each of the quotas, we took the highest daily average resource usage per HTTP request out of the 7-day period:
Multiplied by 5 million requests spread over a 30 day month, these per-request resource statistics translate to daily resource usage of 6.4 CPU-hours and 1.02 gigabytes of outbound data transfer. We top off the numbers by offering 6.5 CPU-hours and 1.07 gigabytes of outbound transfer. Though typically inbound data transfer is a small fraction of outbound data transfer, we made inbound and outbound data transfer symmetric to ease initial data uploads.
Finally — what do we mean by reasonably efficient applications? Simply put, efficient applications avoid unnecessary computation or data transfer, and two techniques common to efficient App Engine applications are the use of caching headers and memcache. Caching headers in an HTTP response prevent a user's browser from needlessly re-downloading information that hasn't changed, both speeding up the user experience and saving bandwidth. Similarly, memcache keeps frequently accessed data in a memory cache on App Engine servers, rather than always reading from disk in the Datastore, therefore saving CPU usage and Datastore load.
Again, these changes ensure we can keep our continuing promise to make it free to get started with App Engine.
What could be better than nine nifty tips and tricks about App Engine? Why, ten of course. As we've been participating in the discussion groups, we've noticed that some features of App Engine often go unnoticed so we've come up with just under eleven fun facts which might just change the way that you develop your app. Without further ado, bring on the first tip:
Although most of the examples show the 'version' field in app.yaml and appengine-web.xml as a number, that's just a matter of convention. App versions can be any string that's allowed in a URL. For example, you could call your versions "live" and "dev", and they would be accessible at "live.latest.yourapp.appspot.com" and "dev.latest.yourapp.appspot.com".
As we alluded to in point 1, App Engine permits you to deploy multiple versions of your app and have them running side-by-side. All the versions share the samedatastore and memcache, but they run in separate instances and have different URLs. Your 'live' version always serves off yourapp.appspot.com as well as any domains you have mapped, but all your app's versions are accessible at version.latest.yourapp.appspot.com. Multiple versions are particularly useful for testing a new release in a production environment, on real data, before making it available to all your users.
Something that's less known is that the different app versions don't even have to have the same runtime! It's perfectly fine to have one version of an app using the Java runtime and another version of the same app using the Python runtime.
It's called the Java runtime, but in fact there's nothing stopping you from writing your App Engine app in any other language that compiles to JVM bytecode. In fact, there are already people writing App Engine apps in JRuby, Groovy, Scala, Rhino (a JavaScript interpreter), Quercus (a PHP interpreter/compiler), and even Jython! Our community has shared notes on what they've found to work and not work on the following wiki page.
The 'IN' and '!=' operators in the Python runtime are actually implemented in the SDK and translate to multiple queries 'under the hood'.
For example, the query "SELECT * FROM People WHERE name IN ('Bob', 'Jane')" gets translated into two queries, equivalent to running "SELECT * FROM People WHERE name = 'Bob'" and "SELECT * FROM People WHERE name = 'Jane'" and merging the results. Combining multiple disjunctions multiplies the number of queries needed, so the query "SELECT * FROM People WHERE name IN ('Bob', 'Jane') AND age != 25" generates a total of four queries, for each of the possible conditions (age less than or greater than 25, and name is 'Bob' or 'Jane'), then merges them together into a single result set.
The upshot of this is that you should avoid using excessively large disjunctions. If you're using an inequality query, for example, and you expect only a small number of records to exactly match the condition (e.g. in the above example, you know very few people will have an age of exactly 25), it may be more efficient to execute the query without the inequality filter and exclude any returned records that don't match it yourself.
Every time you make a datastore request, such as a query or a get() operation, your app has to send the request off to the datastore, which processes the request and sends back a response. This request-response cycle takes time, and if you're doing a lot of operations one after the other, this can add up to a substantial delay in how long your users have to wait to see a result.
Fortunately, there's an easy way to reduce the number of round trips: batch operations. The db.put(), db.get(), and db.delete() functions all accept lists in addition to their more usual singular invocation. When passed a list, they perform the operation on all the items in the list in a singledatastore round trip and they are executed in parallel, saving you a lot of time. For example, take a look at this common pattern:
for entity in MyModel.all().filter("color =", old_favorite).fetch(100): entity.color = new_favorite entity.put()
Doing the update this way requires one datastore round trip for the query, plus one additional round trip for each updated entity - for a total of up to 101 round trips! In comparison, take a look at this example:
updated = []for entity in MyModel.all().filter("color =", old_favorite).fetch(100): entity.color = new_favorite updated.append(entity)db.put(updated)
By adding two lines, we've reduced the number of round trips required from 101 to just 2!
Many people ask about how the datastore will perform once they've inserted 100,000, or a million, or ten million entities. One of the datastore's major strengths is that its performance is totally independent of the number of entities your app has. So much so, in fact, that every entity for every App Engine app is stored in a singleBigTable table! Further, when it comes to queries, all the queries that you can execute natively (with the notable exception of those involving 'IN' and '!=' operators - see above) have equivalent execution cost: The cost of running a query is proportional to the number of results returned by that query.
When adding a new index to your app on App Engine, it sometimes takes a significant amount of time to build. People often inquire about this, citing the amount of data they have compared to the time taken. However, requests to build new indexes are actually added to a queue of indexes that need to be built, and processed by a centralized system that builds indexes for all App Engine apps. At peak times, there may be other index building jobs ahead of yours in the queue, delaying when we can start building your index.
Once a day, we run a task to recalculate the 'Stored Data' figure for your app based on your actual datastore usage at that time. In the intervening period, we update the figure with an estimate of your usage so we can give you immediate feedback on changes in your usage. This explains why many people have observed that after deleting a large number of entities, theirdatastore usage remains at previous levels for a while. For billing purposes, only the authoritative number is used, naturally.
One of the more common and subtle mistakes people make when configuring their app is to forget that handlers in the application configuration files are processed in order, from top to bottom. For example, when installing remote_api, many people do the following:
handlers:- url: /.* script: request.py- url: /remote_api script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py login: admin
The above looks fine at first glance, but because handlers are processed in order, the handler for request.py is encountered first, and all requests - even those for remote_api - get handled by request.py. Since request.py doesn't know about remote_api, it returns a 404 Not Found error. The solution is simple: Make sure that the catchall handler comes after all other handlers.
The same is true for the Java runtime, with the additional constraint that all the static file handlers in appengine-web.xml are processed before any of the dynamic handlers in web.xml.
One anti-pattern that comes up a lot looks similar to this:
q = db.GqlQuery("SELECT * FROM People " "WHERE first_name = '" + first_name + "' AND last_name = '" + last_name + "'")
As well as opening up your code to injection vulnerabilities, this practice introduces escaping issues (what if a user has an apostrophe in their name?) and potentially, encoding issues. Fortunately,GqlQuery has built in support for parameter substitution, a common technique for avoiding the need to substitute in strings in the first place. Using parameter substitution, the above query can be rephrased like this:
q = db.GqlQuery("SELECT * FROM People " "WHERE first_name = :1 " "AND last_name = :2", first_name, last_name)
GqlQuery also supports using named instead of numbered parameters, and passing a dictionary as an argument:
q = db.GqlQuery("SELECT * FROM People " "WHERE first_name = :first_name " "AND last_name = :last_name", first_name=first_name, last_name=last_name)
Aside from cleaning up your code, this also allows for some neat optimizations. If you're going to execute the same query multiple times with different values, you can useGqlQuery .bind() to 'rebind' the values of the parameters for each query. This is faster than constructing a new query each time, because the query only has to be parsed once:
q = db.GqlQuery("SELECT * FROM People " "WHERE first_name = :first_name " "AND last_name = :last_name")for first, last in people: q.bind(first, last) person = q.get() print person
