Previously, developers could only easily get the first 1000-2000 rows of any query or filter easily. This is because BigTable stores data in a way that fetching beyond that was difficult. So if you wanted to get rows 3000-4000 or better yet 3010-3020 for a small page of data, you would have to devise a system to increment the rows previously and then have a marker to filter by that marker, then ensure they were in order. It was troublesome and grew linearly in response times the deeper you needed to go.
With Cursors you can now put in a filter and page through the data, they seemingly encapsulated the difficult part of dealing with their cloud datastore structure and hopefully sped up the deep queried needed in such tools as logs, leaderboards, large lists and well, just about any web app now with any amount of data where you need to page deep. And this will allow faster development of data applications for developers not used to cloud databases as most RDBMS have decent cursor/paging systems.
Wildcard Domains
If you are building a service where people get a username or where you want to shorten urls you can now use wildcard domains. Domains like mybusiness.inventoryserviceapp.com instead of www.inventoryserviceapp.com/mybusinessapp. It feels more personalized and is good for software as a service apps. These are now possible on Google AppEngine.
You can use wildcards to map subdomains at any level, starting at third-level subdomains. For example, if your domain is example.com and you enter text in the web address field:
Entering * maps all subdomains of example.com to your app.
Entering *.private maps all subdomains of private.example.com to your app.
Entering *.nichol.sharks.nhl maps all subdomains of nichol.sharks.nhl.example.com to your app.
Entering *.excogitate.system maps all subdomains of excogitate.system.example.com to your app.
If you use Google Apps with other subdomains on your domain, such as sites and mail, those mappings have higher priority and are matched first, before any wildcard mapping takes place. In addition, if you have other App Engine apps mapped to other subdomains, those mappings also have higher priority than any wildcard mapping.
Note that some DNS providers might not work with wildcard subdomain mapping. In particular, a DNS provider must permit wildcards in CNAME host entries.
Great changes in this update of the engine and built in a scalable way in the cloud. As they refine and learn more about usage there will be more great changes making it less of a burden to move to the cloud, or better yet convincing others to move to Google AppEngine. Other stuff in this build that is great is the AppStats instrumentation apis, custom admin console and more…
No more zip hacks to get all the many files up for Django 1.0 to run on Google AppEngine. GAE now support Django 1.0. You still have to use Google Bigtable models as there is no translation that converts from Django models to Google Bigtable models but 1.0 is so much better to have than .96.
I hope this is a start of supporting and keeping up to date many versions of Django, the Python runtime and common libs so that the 1000 file limits can be used solely for applications running not all the libraries needed.
UPDATE: on the issue for adding Django 1.0 Guido chimed in on 1.1 support when it is out of beta and a nice switch that allows you to choose the django version, hooray! Maybe a python version selector and other libraries soon…
use_library('django', '1.0')
It's indeed fixed, though not yet documented.
To use:
1. Install Django 1.0 into the Python version you're using with the SDK (usually the
system Python).
2. In your app:
from google.appengine.dist import use_library
use_library('django', '1.0')
You no longer need to copy Django 1.0 into your app, nor do you need to mess with
sys.modules, sys.path and zipfiles. (Though those instructions still apply if you
want to use Django 1.1 -- we'll start supporting that once it's out of beta.)
Note, Django 0.96 is still the default, and can also explicitly be requested by using
use_library('django', '0.96') -- in API v2, we'll probably change things so you
*have* to specify a Django version before you can use it.
We're contemplating the support of other open source packages as well (as long as
they don't contain C code).
Cloud based architectures I believe will be more mainstream and used for development even if on self-hosted environments because the data architecture is setup for maximum horizontal scalability from the beginning. BigTable, SimpleDB, HBase, even couchDb type databases will become more prevalent as we move into terabytes and terabytes of data. Data is growing at an alarming rate and this is a natural software evolution to address that.
Great video from Google on building for scale in Google AppEngine. The cloud by nature gives you upwards scaling easily but the data structures and techniques to get data efficiently is much different than RDBMS type optimization. Therefore the more experience and information you can gain on this topic will help you in the future when terabytes of data is the norm.
Google AppEngine blog recently announced the 6 month roadmap. I was hoping an out of beta state was to come soon but looks like it will be a gmail type beta length. But it is still usable and more stable with high availability than anything that s small company can provide. I hope they plan to go live to 1.0 this year, I was hoping the EC2 announcements and official launch of other cloud based offerings would drive them to do that. It makes it much easier to justify using it for production code. I was also hoping for an update to the included django to 1.0.
With that said I digress, the roadmap looks very nice. Jabber/XMPP support, background processesses, sending and receiving and processing inbound and outbound email.
The App Engine team has been plugging away and we’re excited about some pretty big announcements in the near future. In the meantime, we decided to refresh our App Engine roadmap for the next six months with some of the great new APIs in our pipeline:
Support for running scheduled tasks
Task queues for performing background processing
Ability to receive and process incoming email
Support for sending and receiving XMPP (Jabber) messages
As always, keep in mind that development schedules are notoriously difficult to predict, and release dates may change as work progresses. We’ll do our best to update this roadmap as our engineers continue development and keep you abreast of any changes!
I have been saying for some time that cloud based offerings with great tools will win out. Making it simple to setup and manage cloud tools, with tools and hopefully APIs to the tools so that these can be aggreated, will win over hosting clients and be a value add for the cloud providers of today Google App Engine, Amazon AWS, Mosso, Joyent, SliceHost etc.
Today we’re announcing the availability of the Web-based AWS Management Console, which in this first release provides management of your Amazon EC2 environment via a point-and-click interface. A number of management tools already exist: for example a popular Firefox extension known as Elasticfox; however as you read more of this post I believe you’ll agree that the new console is compelling–especially when it’s time to log in as a new AWS developer.
Microsoft launched the Azure cloud based platform at the PDC today. Microsoft has launched many file storage services that were their cloud offerings to date in Mesh, Foldershare, Groove and more. Azure is what appears to be a real cloud platform to compete with Amazon and Google rather than just storage hosting.
Build new applications in the cloud – or use interoperable services that run on Microsoft infrastructure to extend and enhance your existing applications. You choose what’s right for you.
It appears so far that it is pretty Microsoft centric for tool support. Of course the software and servers will be Windows. This week and last, Microsoft platforms have made their way into the cloud platforms at Amazon and now Microsoft. Google also recently announced the support of Java. Another set of aquisitions at Rackspace in the buying of Slicehost and JungleDisk also seem to show the space heating up and the companies all believing in the cloud platform emergence and evolution that seems to be happening.
Amazon EC2 is officially out of beta, it is about time some of these services actually launched. It is hard to convince people to use the cloud layer without being out of beta (AppEngine when’s it gonna happen huh?).
Amazon also launches with windows support, SQL Server support and much more. This is great news in times where budgets are tight and people want to start scalable businesses but want to only pay for what is used. The cloud layer will be a very attractive option to many.
Learn more about the Elastic Computing Cloud (EC2) at Amazon. There are already lots of great simple toos like ElasticFox (Firefox EC2 Extension) to help manage your AMIs from a browser. You can start and stop armies of configured servers from a little extension in your browser.
Developers are getting many tools to build great things. We hope more products are out of beta soon like AppEngine.
AppEngine is getting an update to it’s next available language besides Python (my particular favorite) in Java. Apparently the top candidates were C++, Java and C# support for AppEngine but Java has been added due to the overwhelming library and developer support.
Microsoft also has been playing with entering the cloud with Mesh and now with “Strata”. This would be a cloud for .NET developers.
There are already cloud providers for .NET mainly in Mosso’s offerings of any language being cloud enabled. But one from Microsoft will probably draw all the .NET developers to it like most Microsoft offerings.
This won’t be changing things much for me in the near term but having more platforms available in the cloud and on cloud systems is the natural progression. At some point a platform might have advantages in cheaper processing but for now Python with AppEngine is still the best bet.
New Cursors Eliminate 1000 Row Limit in Datastore
Amazon EC2 is officially out of beta
