New Cursors Eliminate 1000 Row Limit in Datastore

Previously, developers could only easily get the first 1000-2000 rows of any query or filter easily.  This is because BigTable stores data in a way that fetching beyond that was difficult.  So if you wanted to get rows 3000-4000 or better yet 3010-3020 for a small page of data, you would have to devise a system to increment the rows previously and then have a marker to filter by that marker, then ensure they were in order.  It was troublesome and grew linearly in response times the deeper you needed to go.

With Cursors you can now put in a filter and page through the data, they seemingly encapsulated the difficult part of dealing with their cloud datastore structure and hopefully sped up the deep queried needed in such tools as logs, leaderboards, large lists and well, just about any web app now with any amount of data where you need to page deep.  And this will allow faster development of data applications for developers not used to cloud databases as most RDBMS have decent cursor/paging systems.

Wildcard Domains

If you are building a service where people get a username or where you want to shorten urls you can now use wildcard domains.  Domains like mybusiness.inventoryserviceapp.com instead of www.inventoryserviceapp.com/mybusinessapp.  It feels more personalized and is good for software as a service apps.  These are now possible on Google AppEngine.

You can use wildcards to map subdomains at any level, starting at third-level subdomains. For example, if your domain is example.com and you enter text in the web address field:

Entering * maps all subdomains of example.com to your app.

Entering *.private maps all subdomains of private.example.com to your app.

Entering *.nichol.sharks.nhl maps all subdomains of nichol.sharks.nhl.example.com to your app.

Entering *.excogitate.system maps all subdomains of excogitate.system.example.com to your app.

If you use Google Apps with other subdomains on your domain, such as sites and mail, those mappings have higher priority and are matched first, before any wildcard mapping takes place. In addition, if you have other App Engine apps mapped to other subdomains, those mappings also have higher priority than any wildcard mapping.

Note that some DNS providers might not work with wildcard subdomain mapping. In particular, a DNS provider must permit wildcards in CNAME host entries.

[source]

Great changes in this update of the engine and built in a scalable way in the cloud.  As they refine and learn more about usage there will be more great changes making it less of a burden to move to the cloud, or better yet convincing others to move to Google AppEngine. Other stuff in this build that is great is the AppStats instrumentation apis, custom admin console and more…

I hope this is a start of supporting and keeping up to date many versions of Django, the Python runtime and common libs so that the 1000 file limits can be used solely for applications running not all the libraries needed.

UPDATE: on the issue for adding Django 1.0 Guido chimed in on 1.1 support when it is out of beta and a nice switch that allows you to choose the django version, hooray! Maybe a python version selector and other libraries soon…

use_library('django', '1.0')

It's indeed fixed, though not yet documented.

To use:

1. Install Django 1.0 into the Python version you're using with the SDK (usually the
system Python).

2. In your app:
from google.appengine.dist import use_library
use_library('django', '1.0')

You no longer need to copy Django 1.0 into your app, nor do you need to mess with
sys.modules, sys.path and zipfiles.  (Though those instructions still apply if you
want to use Django 1.1 -- we'll start supporting that once it's out of beta.)

Note, Django 0.96 is still the default, and can also explicitly be requested by using
use_library('django', '0.96') -- in API v2, we'll probably change things so you
*have* to specify a Django version before you can use it.

We're contemplating the support of other open source packages as well (as long as
they don't contain C code).

It is still in beta but select developers are being added and developers allowed in will be expanded, I got in as I made the request so it looks like it is still filling up.

Both Java and C# were top requests on AppEngine desired languages/platforms.  They have low level support for specific Google features and infrastructure and wrappers around these features to mimic common java libraries like JPA, JCache and java.net around google urlfetch.  So it is still a bit locked to the platform or changes will need to be made to run on Google AppEngine and other environments.  

Many of the changes made to apps really are due to the scalable nature of the cloud architecture without such things as joins and limits on bulk processing, but in the end these points are really better for scalability from the start.  As for being locked to a platform we are already seeing copies such as AppScale that is an open source implementation of Google AppEngine.  So software architecture, data architecture and how we prepare for the terabytes upon terabytes of shared data in the future might be changing. 

Some unsupported features of major Java libraries…

java.net Features Not Supported

• The URL Fetch service does not support persistent HTTP connections. When the app accesses response data using the URLConnection object, App Engine calls the URL Fetch service to complete the request. After the response data has been accessed, the request data cannot be modified.
• The app cannot set explicit connection timeouts for the request.

Unsupported Features of JPA

The following features of the JPA interface are not supported by the App Engine implementation:

• Owned many-to-many relationships, and unowned relationships. You can implement unowned relationships using explicit Key values, though type checking is not enforced in the API.
• “Join” queries. You cannot use a field of a child entity in a filter when performing a query on the parent kind. Note that you can test the parent’s relationship field directly in query using a key.
• Aggregation queries (group by, having, sum, avg, max, min)
• Polymorphic queries. You cannot perform a query of a class to get instances of a subclass. Each class is represented by a separate entity kind in the datastore.

JCache Features Not Supported

• The JCache listener API is partially supported for listeners that can execute during the processing of a app’s API call, such as for onPut and onRemove listeners. Listeners that require background processing, like onEvict, are not supported.
• An app can test whether the cache contains a given key, but it cannot test whether the cache contains a given value (containsValue() is not supported).
• An app cannot dump the contents of the cache’s keys or values.
• An app cannot manually reset cache statistics.
• Asynchronous cache loading is not supported.
• The put() method does not return the previous known value for a key. It always returns null.

High Level Cloud Provider or Low Level Cloud Provider?

With other cloud providers like Amazon EC2, Mosso, SliceHost and others you can choose your platform.  But Google has an amaxing infrasturcture and taking advantage of BigTable and automatic scaling is nice expecially with a 300MB transfer free quota per app.  There are probably times when each type of provider might be beneficial, lower level like Amazon or higher level like Google AppEngine.

View the announcement here at Google AppEngine Blog which has other great news with the launch:

• Access to firewalled data: grant policy-controlled access to your data behind the firewall.
• Cron support: schedule tasks like report generation or DB clean-up at an interval of your choosing.
• Database import: move GBs of data easily into your App Engine app. Matching export capabilities are coming soon, hopefully within a month.

Cloud based architectures I believe will be more mainstream and used for development even if on self-hosted environments because the data architecture is setup for maximum horizontal scalability from the beginning.  BigTable, SimpleDB, HBase, even couchDb type databases will become more prevalent as we move into terabytes and terabytes of data.  Data is growing at an alarming rate and this is a natural software evolution to address that.

Google AppEngine is finally allowing applications to go beyond the quotas

The pricing for space beyond the free quotas is fairly decent.  About the numbers that were put out earlier around .10 to .15 a GB.

Making the limits larger with a pay version is great for many reasons.  First for Google shareholders and most of all developers that want to develop real applications and services on GAE without the fear of hitting the brick wall when the quotas are up.  But now that wall is a payment.  I was hoping they would lift the quotas a bit more before they rolled in pricing, instead they lowered the quotas…

The pricing for resources beyond those free quotas is:

• $0.10 per CPU core hour. This covers the actual CPU time an application uses to process a given request, as well as the CPU used for any Datastore usage.
• $0.10 per GB bandwidth incoming, $0.12 per GB bandwidth outgoing. This covers traffic directly to/from users, traffic between the app and any external servers accessed using the URLFetch API, and data sent via the Email API.
• $0.15 per GB of data stored by the application per month.
• $0.0001 per email recipient for emails sent by the application

The big bummer in all this is the comment that they are reducing the free quotas:

App Engine remains free to get started. However, along with many performance improvements over the past ten months, we’ve learned that we overestimated our initial free quota values. Therefore, in 90 days we will be reducing the free quota resources. We believe these new levels will continue to support a reasonably efficient application serving around 5 million page views per month, completely free.

The free quota on transfer rate is currently 500GB but will go down to approx 300GB transfer.  This will probably support a 2-5m request app monthly but the old quotas were almost double that.  So it may change some economics that were being planned for GAE.  Let’s hope the free quotas only go up like gmail from now on.

Now we just need full support for full text search, which will become the Google feature benefit I believe, hopefully soon.

The slides from the video

Google AppEngine continues to progress

They have remove limits on high CPU calls, raised the request timeout to 30 seconds, and raised the per file limit to 10MB!

Now we just need it to get out of beta in some format for getting clients loaded up on it.

We’re very excited today to announce that we’ve raised limits on several App Engine operations:

• No more “High CPU Requests”! App Engine Apps were once allowed no more than 2 CPU-intensive requests per minute. We’ve made some adjustments to the way we handle requests, and have eliminated this limitation altogether. To learn more about how this works and the implications for your app, see our documentation.
• Response deadline raised to 30 seconds. The amount of time an App Engine app can take to respond to an incoming request has been raised from 10 to 30 seconds! There are limits on the number of simultaneous active requests an application can process at any given moment–see our docs to learn more.
• Size limits on code files, static files, and requests/responses raised to 10MB! App Engine apps can now receive requests and send responses of up to 10MB in size, and users can upload 10MB code and static files as well. Note that API requests (e.g. memcache.set(), db.put()) are still limited to 1MB in size.

Google AppEngine SDK has been updated

I really hope Google AppEngine can go 1.0 soon.  After the official launch of Amazon EC2 and other services out of beta, mosso, joyent etc it is getting hard to convince people to build on it until it goes out of beta.

Hoping for two things, 1.0 of Google AppEngine and support of Django 1.0 on that release officially. It is possible to run Django 1.0.x on the engine but it is painful using all that disk space (django has lots of files and google has a 1,000 file limit).

http://code.google.com/p/googleappengine/issues/detail?id=61

With that said I digress, the roadmap looks very nice.  Jabber/XMPP support, background processesses, sending and receiving and processing inbound and outbound email.

The App Engine team has been plugging away and we’re excited about some pretty big announcements in the near future. In the meantime, we decided to refresh our App Engine roadmap for the next six months with some of the great new APIs in our pipeline:

• Support for running scheduled tasks
• Task queues for performing background processing
• Ability to receive and process incoming email
• Support for sending and receiving XMPP (Jabber) messages

As always, keep in mind that development schedules are notoriously difficult to predict, and release dates may change as work progresses. We’ll do our best to update this roadmap as our engineers continue development and keep you abreast of any changes!