Last year, we featured a guest post by Jeffery Rosen, of Wolfire Games about the Humble Indie Bundle, an awesome promotion for a good cause: spreading awareness of indie games, and supporting charity. For 12 days, you could name your own price for a bundle of awesome independent games, with proceeds split as you wish between the game developers and two deserving charities, Child's Play and the Electronic Frontier Foundation. They hosted the Bundle on App Engine, to great effect - read the original blog post to learn more.
The good news is, they're doing it again, with the same App Engine infrastructure, and a new set of games! There's only 4 days left, so be sure to check them out if you're interested.
App Engine version 1.4.0. is here! It’s our most significant release of the year for the App Engine SDK, including a number of very big features that we know developers have been eagerly awaiting:
As well, we’ve spent a lot of time this release on reducing or removing the limitations of some of App Engine’s existing APIs
As you can imagine, some of these changes drastically expand the scope of applications that can be easily built using App Engine so download the SDK while it’s hot!
Keep an eye out for more blog posts on how you can take advantage of the new features to build your apps very soon. And we’ve got a few more big features coming very soon, such as a High Replication Datastore, so keep an eye on the App Engine roadmap and stay tuned.
Over here at the UCSB Racelab, we've complained endlessly about finding a web framework we actually could use. For a long time we thought we just wouldn't be able to find it - many were so-so or good but only after a substantial learning curve. So imagine our surprise back in April 2008 when we heard about what we thought would be just-another-web-framework provided by Google in the Python version of App Engine. But after giving it a try, we were smitten. We finally found a web framework that (1) we could actually use on non-trivial projectsand (2) we could teach in nine-week classes without having students lose half the time with the idiosyncrasies of the programming language involved or the web framework itself. Furthermore, the minimalistic APIs make it simple to get work done: it did for us exactly what we needed and nothing else.
Yet as researchers and hackers-at-heart there was one thing that we really wanted to do with App Engine that we couldn't do: run it on a whole bunch of our machines and tinker with it. A similarly-minded hacker named Chris Anderson had released AppDrop, which was a modified version of the App Engine SDK that hooked up to PostgresSQL and run in Amazon EC2, but only ran over a single machine. So after much discussion, we came up with the following short list of things we wanted to do with App Engine:
So with that in mind, we created AppScale, an open-source cloud platformfor Google App Engine applications. Here's how we did it:
We took the standard three-tier web deployment approach and clearly segmented each tier into a specific component in the system: an AppLoadBalancer routes users to their applications, an AppServer runs the user's App Engine app, and an AppDB handles database interactions. Each have clearly defined roles in the system and are controlled by an AppController, a daemon that runs on each machine, monitors each component, and controls the specific order in which services are started. It writes all the configuration files for each service and coordinates services between the other AppControllers in the deployment. For those interested, we detail the specifics on the original AppScale implementation in this paper.
We also wanted to embody the principle of "standing on the shoulders of giants", and as such, we employ open-source software as often as possible, where appropriate. Our AppLoadBalancer employs the nginx web server as well as the haproxy load balancer to ensure high performance. Our Memcache API implementation uses memcache under the hood, while our MapReduce API uses Apache Hadoop, which we added to give App Engineusers running over AppScale the ability to run Hadoop MapReduce jobs from within their web applications.
Because we were able to keep the database support abstracted away from the other components in the system, we were able to add support for nine different data storage solutions within AppScale: HBase, Hypertable,MySQL, Cassandra, Voldemort, MongoDB, MemcacheDB, Scalaris, and SimpleDB. Many of these databases have seen interest in recent years but have been hard to measure under comparable conditions, and vary greatly. To give a few examples, they vary in the query languages they provide, their topologies (e.g., master / slave, peer-to-peer), data consistency policies, and end-user library interfaces. This has made it non-trivial for the community to objectively determine scenarios in which one database performs better or worse than another and investigate why, but under AppScale, deploying all these databases is done automatically with no interaction from the user. And because AppScale is open-source, if a developer doesn't like the particular interface we use for a database, they can improve on it and give back to the community. We've usedAppScale internally to evaluate the performance of Google App Engine applications on these datastores as well as developed an App Engine app, Active Cloud DB, that exposes a RESTful API that developers can useto access these datastores from any programming language or web framework.
Finally, the most important lesson we learned was the value of incremental development. Our core development team fluctuates between two to three developers, so from the first meeting we had, we knew thatour very first release couldn't support every App Engine API nor could it run nine databases seamlessly. Therefore, we started off with support for the two BigTable clones, HBase and Hypertable, as well as supportfor just the Datastore API, the URL Fetch API, and the Users API within App Engine. From there, we learned what datastores people actually wanted to see support for as well as what APIs people wanted to use. Wewere also able to add APIs within App Engine apps deployed to AppScale to be able to run virtual machines under the EC2 API, while also running computation under the MapReduce API.
But developing AppScale was certainly not a cakewalk for us. Over the course of the last two years, five major issues (some technical and some not) have arisen within the project:
All of these problems are greatly exacerbated by only having a two-to-three person core developer team, but this also makes the AppScale project particularly interesting to work on. Despite having worked on AppScale for two years, there are still tons of interesting problems to work on and we still love the Python App Engine web framework as much as we did when we first picked it up. And of course, AppScale is open-source, under the New BSD License, so feel free to download it and tinker around like we have! Check out AppScale at:
http://appscale.cs.ucsb.edu
http://code.google.com/p/appscale
Today, we’re releasing version 1.3.8 of the App Engine SDK. Whether you’re a Java or a Python developer, this release includes several exciting new features for improving monitoring, performance, and maintenance tasks.
This release includes a new page in the Admin Console, called the Instances page. This page allows you to view information about all server instances currently in use by your application. This information can be useful in debugging your application and also understanding its performance characteristics. There’s no configuration needed for this feature. Just click the “Instances” link on the left hand navigation of the Admin Console to see Average QPS, latency, and memory for an instance.
This release also has a couple new Task Queue features: First, the maximum bucket size that you can specify during queue configuration is now 100, up from 50. Second, we’ve added a new "Run Now" button to the Task Queues section of the Admin Console that enables developers to run a task immediately. This can be very helpful for debugging your tasks in production.
This release contains a new feature for Python apps: builtin handlers that allow you to quickly and easily enable standard functionality in your application without adding additional code to your codebase. The libraries available today are remote_api, appstats, and the datastore_admin feature (see below). For example, to use the remote_api with your application, simply add the following to your app.yaml file:
remote_api
appstats
datastore_admin
app.yaml
builtins:- remote_api: on
If you are already using the remote api endpoint your app, you can choose to remove the entry in the handlers section of your app.yaml and use the above directive instead to simplify your app.yaml file.
Support for builtin handlers is not yet available for Java applications, but will be available in an upcoming release.
Note: this feature is currently only available by default for Python; see the note below for ways to use it with Java application.
Today, we are releasing an experimental addition to the admin console which provides a simple UI for delete all entities, or all entities of a given kind, in your datastore. To enable this functionality, simply enable the following builtin in your app.yaml file:
builtins:- datastore_admin: on
Adding these lines to app.yaml enables the “Datastore Admin” page in your app’s Admin Console, where you can see all of the entity types you are able to delete:
Be aware that these deletes are issued by your application (you can read about how the handler works by looking at this code file in the SDK). For this reason, your application will use resources, most significantly CPU, for the deletions you issue which will count towards your application’s daily resource budget.
Datastore delete is currently available only with the Python runtime. Java applications, however, can still take advantage of this feature by creating a non-default Python application version that enables Datastore Admin in the app.yaml. Native support for Java will be included in an upcoming release.
Finally, the python pre-compilation feature we announced in 1.3.5 is now turned on for all new python application uploads using the 1.3.8 SDK by default. If you wish to disable this feature, just specify the flag --no-precompilation on the appcfg.py command line when uploading your app.
--no-precompilation
appcfg.py
This release also contains a few more small features and bug fixes. You can read about the full release in our release notes in Python and Java. As always, your feedback in the forums is appreciated (and had a significant influence on this release!).
Last week we announced our 1.3.6 release with lots of new, exciting features to try out. After it was released, developers dug into all the new functional and discovered a pair of issues that required a quick fix. Today we are releasing version 1.3.7 of the SDK in both Java and Python to fix two issues that shipped with last week’s release 1.3.6.
The first issue was with the Python SDK, which was missing the functionality to assign a namespace based on the Google Apps domain. With this release the function google_apps_namespace() should exist and function as documented.
The second issue we fixed was with the Java SDK. The getServingUrl() function for the new dynamic image resizing service was throwing a SecurityException in the SDK. This issue has now been fixed and this function should work as expected.
Thank you to all the people who reported these issues. Please visit the download page to get the new version of the SDK.
Today marks the 1.3.6 release of App Engine for Java and Python. In this release we have made available several exciting new features, relaxed quota and datastore limitations, and added various issue fixes.
We are pleased to announce support for multi-tenancy for applications via the Namespaces API. With multi-tenancy, multiple client organizations (or “tenants”) can all run the same application, segregating data using a unique namespace for each client. This allows you to easily serve the same app to multiple different customers, with each customer seeing their own unique copy of the app. No changes in your code are necessary to use this API-- just a little extra configuration. Further, the API is also designed to be very customizable, with hooks into your code that you can control, so you can set up multi-tenancy in any way you choose.
Check out our application examples for Java and Python to demonstrate how to use the Namespaces API in your application. The API works will all of the relevant App Engine APIs (Datastore, Memcache, and Task Queues). Check out our docs for Java and Python to learn more.
This release also includes a new, high-performance image serving system for your applications, based on the same infrastructure we use to serve images for Picasa. This feature allows you to generate a stable, dedicated URL for serving web-suitable image thumbnails. You simply store a single copy of your original image in Blobstore, and then request a high-performance per-image URL. This special URL can serve that image resized and/or cropped automatically, and serving from this URL does not incur any CPU or dynamic serving load on your application (though bandwidth is still charged as usual). It’s easy to use, just call the Python function get_serving_url, or the Java function getServingUrl and supply a Blob key (with optional serving size and/or crop arguments), and you can now serve dozens or hundreds of thumbnails on a single page with ease. To enable high performance image serving in your deployed application, you'll need to enable billing.
Since launch, many developers have asked to be able to serve custom error pages instead of those automatically served by App Engine. We are happy to announce today we are supporting static HTML error pages that can be served for you automatically for over quota, DoS, timeout and other generic error cases, that you previously could not control. You can configure custom error handlers in your app.yaml or appengine-web.xml file. Check out the Java or Python docs for more information.
We have also continued the trend of lifting some system limitations that have been in place since launch. The Datastore no longer enforces a 1000 entity limit on for count and offset. Queries using these will now safely execute until they return or your application reaches the request timeout limit. Also, based on your feedback, we have raised nearly all of the burst quotas for free apps to the same level as the burst quotas for billed apps. Check out the docs for more information on quota limits.
In addition to all of the new features, we’ve included several bug fixes which you can read all about in the release notes for Java and Python.
-- Posted by the App Engine team
We recently announced the Mapper API, a first step in a broader effort to provide full MapReduce capabilities on App Engine. While we still have some work ahead of us, there’s already a lot that can be done with today’s Mapper API. One area of particular interest is report generation.
Most applications create and maintain large numbers of detail records: entities in data models, transaction history or event logs. In order to glean useful information out of these vast data sets, your application has to iterate over your entities, summarize and breakdown the results. That’s where the Mapper API comes in.
The Mapper API uses the Task Queue to enable your application to rapidly iterate over its data sets, whether small or very, very large. The API takes care of the tedious bookkeeping involved in efficiently scheduling and keeping track of all those tasks. The Task Queue, in turn, automatically ‘pushes’ the work to your app.
Christopher O’Donnell (@markitecht), a technology product designer and developer from Cambridge, Massachusetts, was kind enough to share with us some of the motivations and implementation details behind his own project. In a new guest article, Modern Funnel Analytics Using Mapper, Christopher makes extensive use of the Mapper API to illustrate the process of generating rollup reports. With his approach, he's able to provide both summarized results and drill down capabilities.
If you like Christopher’s article, you should also checkout out the many other interesting articles on the Google App Engine home page. And, if you have an interesting App Engine article or story you'd like to share, let us know. We'd love to hear about it.
Posted by Fred Sauer, App Engine Team
"As you know, we were massively against the clock with the launch of the cycle hire scheme, and we needed something we could get going with fast that would effortlessly scale to perhaps tens of thousands of mobile users. App Engine seemed the perfect choice from what we had read of it before the meeting, and after your presentation it was obviously the way to go. Your recommendation to use Python was scary given neither of us knew a thing about it, but then again we only knew Java from Android not from web development so we didn't have the domain knowledge of building Java web services. So we went with Python, and it worked out really well. I'm astounded how we actually delivered this product in a very short space of time when we both have full schedules working on projects for our clients and other demanding outside interests. Particularly satisfying was having a solution that was agile and flexible enough to enable us to display live cycle availability data within hours of it becoming unexpectedly available at the launch, so we were live in the field with real-time data that very same launch morning, a feature our competitors are still struggling to replicate."
"The single coolest thing about this project is that it was possible to go from a state of knowing nothing whatsoever about App Engine or Python (other than the mile-high view) to having a working and useful application inside of eight hours. We're long-time geeks but we're not geniuses. For us to pick up a new language, a new SDK, a new environment, a new way of doing things, and produce anything of value at all in such a short time speaks volumes about the value, potential, and quality of App Engine and Python.After installing the App Engine SDK, yes, the very first thing I did was your online tutorial. I did "Hello World" to find my feet then continued into webapp, since a clean URL handler with easy ways to get at HTTP variables seemed essential. Then I immediately jumped into learning about data storage. And wow, what an enlightenment that turned out to be! Goodbye SQL, don't think I'm going to miss ya.... :-)Since the app's purpose is to manage just ~400 simple objects representing Cycle Hire Stations, each of which contains only Plain Old Data types — no object references or anything possibly messy — I felt I knew enough to implement it now, and so I dived in. And it was so easy! I started with a handler to rebuild the datastore from scratch. Then I wrote a "get" type of handler to retrieve information about groups of hire stations (returning the data in JSON). Finally I wrote an "update" handler so that updated information about cycle hire stations could be posted, and that was it. Job done.One thing that initially confounded me was an HTTP 500 error caused by our "reset" handler exceeding the 30-second request limit. For a while I was ready to despair; HTTP 500s to anyone with much ASP experience usually means a hideous low-level bug somewhere! However, once we discovered the problem, this was easy to fix by splitting the work into multiple requests (/reset1, /reset2, etc.) It's an admin function that only we'd ever be using, so no harm done and no need to work out anything more clever.I know we've barely scratched the surface of what can be done with App Engine. We've yet to use Memcache, background tasks, batched updates, or anything beyond simple cloud-based data storage. But that simple thing alone seemed then, and still seems, not far short of miraculous. To not have to worry about databases, servers, uptime, upgrades and above all scaling... to not have to think about any of that at all is such an immense freedom. I'm completely hooked on it and am unlikely to go back to my traditional server tools of MySQL and PHP.
"We formed Little Fluffy Toys Ltd as a vehicle for Android development where we do consultancy work as well as our own stuff like the Cycle Hire Widget and Social Wallpaper. Whilst all custom development enquiries are very welcome, we're also interested in hearing from people or organisations that would like us to customise Cycle Hire Widget for their particular domain, whether it's cycles with availability in another city, coffee shops with opening hours in a geographic region, or dieting group meetings at pertinent times nearby. You name it, there are a gazillion applications for it!"
Back in March, we announced that all App Engine apps hosted on appspot.com would be reachable over IPv6, from ISPs enrolled in the Google over IPv6 program.
Today, we're adding the same service to all domains hosted by ghs.google.com. If your App Engine app runs on your own domain name using Google Apps, then be sure to review our earlier blog post, which discusses the caveats of parsing client IP addresses, and how to use the SixXS Gateway to test your site over IPv6.
Currently, the Google over IPv6 program presents AAAA records only to ISPs within our whitelist. This helps ensure that our services remain reliable, by reducing exposure to clients with broken IPv6 connectivity. As the maturity of IPv6 deployment improves, we expect this whitelist to grow, eventually disappearing as it encompasses the entire Internet.
That said, we know that some of you are fans of IPv6 who would prefer to skip the whitelist, and offer dual-stack service to everyone immediately. To make your domain bypass the whitelist, just change your CNAME pointer from ghs.google.com to ghs46.google.com. Please note that, while ghs46 will cause your App Engine site to receive a larger fraction of its traffic over IPv6, our studies show that it may make your site slow or unreachable for up to 0.1% of clients across the Internet, so use this alternate domain with care.
As before, if you experience any problems with IPv6 serving and App Engine, please report it in the App Engine issue tracker or post about it in the App Engine discussion group.
Posted by Paul + the App Engine Traffic Team
If you have been running an App Engine application over the last few months, you will have noticed that performance of the Datastore has drastically improved. In fact, it quickly improved after the blog post we wrote in May acknowledging the performance issues. We chose to wait until this point to ensure that we could maintain the performance and quality of service. Thanks to a huge effort from the Datastore engineering team and our Production team, the metrics over the past month have made us confident that we have returned to a level of performance that we are proud to offer. With that in mind, we are now going to re-enable Datastore CPU usage billing.
On Tuesday, August 3rd, we will re-enable charging for Datastore CPU usage.
This does not mean that we are stopping work on Datastore performance and reliability-- quite the opposite, in fact. The issue has forced the team to step back and prioritize our long term roadmap for the Datastore. While we’re happy with our performance now, there are some very large changes to the Datastore in the pipeline that will continue to improve the App Engine platform. These include features such as a more highly replicated Datastore (now part of the roadmap) and improvements to the index building pipeline.
Thank you all for your patience and continued use of App Engine during this time. We’re excited to continue to serve our apps’ continued growth.
At Google I/O we announced the Mapper API. Built completely on top of public App Engine APIs today, this API is only the first component of App Engine’s MapReduce toolkit, but can be extremely useful on its own.
The Mapper API can already be of use to many developers who would otherwise need to build their own tool for doing large scale data manipulation. In addition to taking care of the distribution of these jobs over task queues, it provides the ability to store state, batch datastore writes via mutation pools, and ships with an easy to use administrative interface for job management, all optimized for the constraints of App Engine’s dynamic serving environment. Some examples of the types of operations that work with minimal configuration with this tool:
When you’re ready to jump in and start using the tool, head over to the project homepage on Google Code. You’ll want to check out the “Getting Started” page for the language you’re using:
Happy Mapping!
- Fred, Mike, Ikai, Nick + the App Engine team
Fresh from our exciting news from Google I/O last quarter (launch of Google App Engine for Business and new product developments), we now look ahead to the mid-summer months. Since I/O, we've already appeared at a number of events including the Google DevFest Australia and Philippines, PyCon AU, and the Malaysia Open Source Conference, with many more coming this month. There are several well-known large conferences (and many smaller ones too) that our team members will be attending and/or speaking at, so hopefully we'll get to meet you at one of them!
In addition to the ones listed above, Google will be hosting more DevFests in Asia and South America later this year. Keep an eye out for specific dates and locations on the Google DevFest home page. There are also 5 larger global Google Developer Day events in the fall. We would love to meet with App Engine developers at a DevFest, Developer Day, or any of the other conferences above!
See you around the world!
Posted by Wesley Chun, Google App Engine Team
We have two scheduled upcoming maintenance periods for the Datastore:
As part of our on going Datastore reliability work, we'd like to inform developers that during the period between the two maintenance events listed above, we are expecting that Datastore performance will be impacted and applications will see higher read/write latencies. This maintenance and the higher latency in between are an unfortunate, but necessary step in our long-term plan towards completing the work to address the datastore latency issues (as described in our recent blog post). We recognize the inconvenience to applications and their users caused by this. For this reason, we continue to offer free datastore CPU usage until we are confident we have cleared these issues.
You can read more about the currently scheduled maintenance periods on the notification to the Downtime Notify list. As a reminder, be sure to sign up with the Downtime Notify list for alerts and updates on scheduled (or unplanned) maintenance, and follow @app_engine on Twitter for ongoing announcements.
Today we are happy to announce the 1.3.5 release of the App Engine SDK for both Python and Java developers.
Due to popular demand, we have increased the throughput of the Task Queue API, from 50 reqs/sec per app to 50 reqs/sec per queue. You can also now specify the amount of storage available to the taskqueue in your app, for those with very large queues with many millions of tasks. Stay tuned for even more Task Queue scalability improvements in the future.
Additionally, in this release we’ve also added support for precompilation of Python source files to match the same feature we launched for Java last year. For Python, you can now use precompilation to speed up application loading time and to reduce CPU usage for new app instances. You can enable precompilation by including the following lines in your app.yaml file:
This will start offline precompilation of Python modules used by your app when you deploy your application. Currently precompliation is off by default for Python applications, but it will be enabled by default in some future release. (Java precompilation has been enabled by default since the release of 1.3.1.)
To give you a taste of what this feature is like, we tested this on a modified version of Rietveld (which included a copy of Django 1.0.4 in the app directory, and which did not use the datastore in its base url). The latency and CPU usage results for the initial load of the application, after uploading a new version of the app and requesting the homepage, were:
Of course, any individual app’s performance will vary, so we recommend that you experiment with the setting for your application. Please submit your feedback and results to the support group!
In addition to Task Queues and Python precompilation, we have made a few changes to the Blobstore in 1.3.5 We have added file-like interfaces for reading Blobs. In Python, this is supported through the BlobReader class. In Java, we have implemented the BlobstoreInputStream class, which gives an InputStream view of the blobs stored in Blobstore.
More information on this release can be found by reading the release notes for Python and Java.
// Obtain the credentials from your configs credentialObj = new APICredential();credentialObj.setAPIUsername(getServletConfig() .getInitParameter("PPAPIUsername"));credentialObj.setAPIPassword(getServletConfig() .getInitParameter("PPAPIPassword"));credentialObj.setSignature(getServletConfig() .getInitParameter("PPAPISignature"));// setup your AppID from X.comcredentialObj.setAppId(getServletConfig() .getInitParameter("PPAppID"));// setup your Test Business account email // in most cases this would be associated with API CredentialscredentialObj.setAccountEmail(getServletConfig() .getInitParameter("PPAccountEmail"));// add required error condition checks//....
try { // CreateSimplePreapproval request to setup a simple // preapproval with no Payment Period Set CreateSimplePreapproval simplePreapproval = new CreateSimplePreapproval(); // set the API Credentials object (as given in the code above) simplePreapproval.setCredentialObj(credentialObj); // starting date in yyyy-MM-dd format simplePreapproval.setStartingDate("2010-06-25"); // ending date in yyyy-MM-dd format // in this case it's for an year from the starting date simplePreapproval.setEndingDate("2011-06-25"); // set max total amount of all Payments simplePreapproval.setMaxTotalAmountOfAllPayments(52.00); // set max amount for each payment - let's say $1 simplePreapproval.setMaxAmountPerPayment(1.00); // set max number of payments allowed - (52 weeks) simplePreapproval.setMaxNumberOfPayments(52); // set where to send the user in case of a cancellation simplePreapproval.setCancelUrl(req.getRequestURL() + "?action=preapproval&cancel=1"); // set where to return the user after successful approval simplePreapproval.setReturnUrl(req.getRequestURL() + "? return=1&action=preapproval&preapprovalKey=${preapprovalKey}"); /* Set other required fields */ // ... // // set memo for user's transaction history simplePreapproval.setMemo("Preapproval for GAE Sample"); // send the request PreapprovalResponse preapprovalResponse = simplePreapproval.makeRequest(); } catch (//...exceptions go here...//) { // handle exceptions as necessary}
This is a guest post from Jeffrey Rosen, of Wolfire Games.
Recently, Wolfire Games, joined by four other independent video game developers and two charities, put on a pretty unusual promotion: The Humble Indie Bundle. For twelve days, you could have gone to the promotional site and paid what you want (divided any way you choose among the developers, the EFF, and the Child's Play Charity) for a bundle of independent video games, all compatible with Mac, Windows, and Linux.
The reason why I've been invited to write this guest blog post is because the site was built on Google App Engine and, according to Google Analytics, it served about 3.4 million pageviews (1 million unique visitors) in the 12 days it was live. It grossed over $1.2 million. Depending on who you ask, this is quite a lot of traffic, and we exceeded the free App Engine quota. Google sent us a bill for a grand total of $71.56.
I've been a long time advocate of Google App Engine on the Wolfire Blog, even when traffic was tiny, for a number of reasons. I really like its ease of use, and the fact that Google engineers, including Guido himself (the creator of Python), are the ones making sure everything works at 4 AM, and Google's infrastructure is responsible for data durability and security. The pay-as-you-go price combined with a generous free quota is icing.
As an independent video game developer, it's pretty crazy to have these resources at my disposal. While I'm confident in my Python web coding ability, I am definitely not confident in being a Linux sys-admin and commiting to an expensive server that may or may not handle the load properly (especially scary when critical data is involved). This upfront risk and steep learning curve would probably have been the straw that broke the camels back and turned the bundle into vaporware.
According to Google Analytics we did about 3.4 million pageviews (1 million unique visitors) during the bundle period on the Wolfire site (heavily concentrated around the first and last days). The traffic came in huge surges from Reddit, Slashdot, Digg, and a plethora of gaming and tech sites, and people seemed to really enjoy pushing the server by refreshing the page to update the real-time statistics. App Engine performed as advertised: additional servers were seamlessly brought up as needed and shut down when appropriate.
One thing that was really neat about the bundle is that it had a bunch of real-time statistics about purchases along with a leaderboard of the top contributions. There were many interesting donations, for example, $3333.33, $1337, $327.67, and $313.37 and watching the average donation go up and down was really addictive. I think being able to refresh the page to see the sales coming in was a really big draw for the bundle. This feature was really easy to implement on App Engine using a sharding counter. I'm really looking forward to native comet support in App Engine so the next bundle can have this update completely live, without polling or refreshing. Memcache and the datastore is built for stuff like this, so it was very fast.
One awesome thing about App Engine is how it handles the deployment for you transparently and seamlessly. I felt comfortable developing, testing, and ultimately deploying new features to the site, even while it was serving say, 100 visitors per second. App Engine handles all the details of seamlessly bringing down the servers and getting the new ones in place even under intense loads. Because of this, I was able to hack on new features to the site as necessary. For example, I expanded the statistics so that you could see the platform specific average payments in real-time. That feature was started, tested, and deployed on maybe the third day of the bundle. Similarly, I spontaneously added an unplanned sixth game (Samorost 2) to the bundle on the fifth day and deployed minor tweaks dozens of times throughout the promo.
One thing that the bundle made clear to me is that raw Python code is much faster than you might expect. After I finished building the prototype of the bundle, I was getting ready to heavily optimize everything: each hit to the Humble Bundle page executed a few hundred lines of my Python code, doing all sorts of calculations for the statistics and inserting that into a massive Django template. Initially I thought that there would be no way this would be scalable and I'd need to aggressively cache the whole page. However, when I looked at the estimated CPU usage (tip: if you look at the request headers of a page while logged in as an admin, App Engine gives you some estimates on how much it will charge you for 1000 requests and how long it will take) it was something like 0.00375 CPM and was executing in milliseconds. Despite what I had assumed to be inefficient, unoptimized code, the CPM was so ridiculously low and it was executing so fast, I decided not to even bother optimizing the backend further. The only thing that was taking any time at all were the few memcache calls to retrieve the real-time list of tweets, real-time raw sales data, and other interesting stats. In hindsight though, I could have easily shaved off 30 ms by combining all the memcache requests into a single RPC with memcache.get_multi.
While I am really satisfied with App Engine and credit its existence and my familiarity with it for the existence of the Humble Indie Bundle in the first place, there were a few gotchas:
Due to a scheduled maintenance period on the datastore, the bundle was down for about 20 minutes (we need to store metadata about people's orders before we forward them to PayPal, Amazon Payments, or Google Checkout). Normally a message pops up to inform people when downtime occurs is going on, but this didn't activate for some reason, so we had to handle a lot of email about a blank page when people were trying to place their orders. I understand that Google is addressing this though.
The features that I would like to ask for are mostly all "on deck" on the roadmap: SSL for custom domains, comet support, and longer background tasks in particular. I'd also like to see App Engine focus on utilizing Google's infrastructure by serving non-static requests geographically like "web acceleration" services from CDNs. There are some trade-offs due to the nature of the datastore, but I think it can be done.
Thanks for having me on the blog! People can find me on the Wolfire Blog where we talk about more technical and game design topics, and check out our upcoming game Overgrowth.
