googlecloudplatform-test: March 2014

Updates to Google BigQuery following Cloud Platform Live

March 31, 2014

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.announcedGoogle BigQuery1000x increase in streaming capacitywe announced the ability to stream datalive demoTable wildcard functionsTABLE_DATE_RANGE()documentation and syntax for table wildcard functionsImproved SQL support and table viewsGoogle Cloud SDK)User-defined metadataJSON parsing functionsJSON_EXTRACTJSON_EXTRACT_SCALARJSONPathSELECT json_extract_scalar(
"{'book': {
'category':'fiction',
'title':'Harry Potter'}}",
"$.book.category");
Fast parallel exportsexporting data documentationMassive price reductionsprice reductionR has its own connectortutorialPython pandas toovideo we made with PearsonActiveRecord connectorlogs with fluentdFelipe Hoffa

Announcing Google Cloud DNS

March 28, 2014

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.Google Cloud DNS Domain Name SystemGoogle Compute EngineGoogle Cloud Storage zonesrecordshttp://developers.google.com/cloud-dnsGoogle Developers Console Screen Shot 2014-03-24 at 2.21.14 PM.png

Screen Shot 2014-03-24 at 2.21.14 PM.png

Bringing together the best of PaaS and IaaS

March 27, 2014

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.Managed Virtual MachinesDeployment Manager

Managed Virtual MachinesGoogle Cloud Platform LivManaged Virtual MachinesGoogle Compute EngineGoogle App Enginevm:truehereIntroducing Google Cloud Deployment Manager Google Cloud Deployment Manager

The attributes of any Compute Engine virtual machines (e.g. instance type, network settings, persistent disk, VM metadata).

Health checks and auto-scaling

Startup scripts that can be used to launch applications or other configuration management software (like Puppet, Chef or SaltStack)

Deployment ManagerUpdate on operating systemsSuSE and Red Hat Enterprise Linuxsign up for accessSimple, lower pricesmentioned on Tuesday

We’ve dropped prices by 32% across the board.

We’re introducing sustained-use discounts, which lower your effective price as your usage goes up. Discounts start when you use a VM for over 25% of the month and increase with usage. When you use a VM for an entire month, this amounts to an additional 30% discount.

only pay for the minutes that you use

dedicated memcacheDatastoreSNI SSLPageSpeed hearing from youhere

New developer experiences for Cloud Platform

March 26, 2014

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.Google Cloud PlatformIsolating production problems fasterGoogle App Engine

You can see here where Brad introduced an error into production.

Screen Shot 2014-03-21 at 1.36.21 PM.png

Shortening your time to fix

Fix simple problems directly in the browser

Trigger your push-to-deploy setup instantly

An integrated ecosystem of tools you knowGit push-to-deployo does the Google Cloud Platform team

Push-to-deploy now supports Java builds

The release history showing build, test and deployment status

Google Cloud SDKrecently posted gcloudinit

Google Cloud Platform Live - Blending IaaS and PaaS, Moore’s Law for the cloud

March 25, 2014

Updated at 12:52 PM PST to include App Engine pricing information
Editors note: Tune in to Google Cloud Platform Live for more information about our announcements. And join us during our 27-city Google Cloud Platform Roadshow which kicks off in Paris on April 7. Google Cloud Platform LiveCloud PlatformIndustry-leading, simplified pricingMoore's Law

Compute Engine reduced by 32% across all sizes, regions, and classes.

App Engine pricing is drastically simplified. We've lowered pricing for instance-hours by 37.5%, dedicated memcache by 50% and Datastore writes by 33%. In addition, many services, including SNI SSL and PageSpeed are now offered to all applications at no extra cost.

Cloud Storage is now priced at a consistent 2.6 cents per GB. That’s roughly 68% less for most customers.

Google BigQuery on-demand prices reduced by 85%.

Sustained-Use discountsSustained-Use Discounts

Sustained-Use Discounts automatically reward users who run VMs for over 25% of any calendar month

Making developers more productive in the cloud

Build, test, and release in the cloud, with minimal setup or changes to your workflow. Simply commit a change with git and we’ll run a clean build and all unit tests.

Aggregated logs across all your instances, with filtering and search tools.

Detailed stack traces for bugs, with one-click access to the exact version of the code that caused the issue. You can even make small code changes right in the browser.

Introducing Managed Virtual Machines Managed VMsExpanded Compute Engine operating system supportWindows Server 2008 R2Red Hat Enterprise LinuxSUSE Linux Enterprise ServerReal-Time Big DataBigQuery Streamingreserved query capacityConclusionUrs Hölzle

MaestroDev taps Compute Engine to provide cost-effective task management for DevOps environments

March 24, 2014

Today’s post comes from Brett Porter, CTO of MaestroDev, a DevOps Automation service for enterprise-class companies. MaestroDevGoogle Compute EngineSub-Hour BillingRightScale

Google Cloud Platform and Leanplum help app developers conduct on-the-fly A/B tests

March 21, 2014

Editor’s note: Today’s guest blog comes from Andrew First, co-founder and CTO at Leanplum, a platform for optimizing the mission-critical metrics of mobile apps. Leanplum processes billions of events per day to provide a faster and more effective way for developers to optimize mobile applications. LeanplumGoogle Cloud PlatformGoogle App EngineGoogle BigQueryGoogle Cloud DatastoreGoogle Cloud Storage Google Compute Engine

By default, App Engine indexes every field in your datastore entities. By indexing only the fields that you need, you can reduce datastore costs by an order of magnitude.

Run a separate version of your application as a staging instance, so you can test out new features. You can run multiple versions of your app simultaneously, so that every engineer has their own application running at the same time to test new features or to show demos.

Use AppStats to find bottlenecks in your application. Using AppStats, we were able to reduce the latency of our API by an order of magnitude.

Use Traffic Splitting to roll out risky changes. Fittingly, we are an A/B testing company that uses A/B testing on our own backend. When we roll out substantial backend changes, we can test them out on a small percentage of traffic to make sure it works well, and to measure the impact on latency and number of instances used.

Use BigQuery to analyze your logs. We wrote a custom mapreduce to collect our logs and dump them into BigQuery. This gives us the ability to gather statistics on latencies for different RPC calls, frequencies and types of errors, and to resolve customer issues by being able to query over our logs in seconds.

Cassandra Hits One Million Writes Per Second on Google Compute Engine

March 20, 2014

known for creating scalable high performance systemsone million requests per secondCassandraGoogle Compute Engin

sustain one million writes per second to Cassandra with a median latency of 10.3 ms and 95% completing under 23 ms

sustain a loss of ⅓ of the instances and volumes and still maintain the 1 million writes per second (though with higher latency)

scale up and down linearly so that the configuration described can be used to create a cost effective solution

go from nothing in existence to a fully configured and deployed instances hitting 1 million writes per second took just 70 minutes. A configured environment can achieve the same throughput in 20 minutes.

Cassandra Quorum commitSetup

setup instructionsResults Cassandra Blog Post - latencies and throughput (3).png

Cassandra Blog Post - Latency and Throughput superimposed.png

Conclusionone million Cassandra writes per second at a cost of $0.07 USD per million writes

Optimizations to Google Cloud Platform for Real Time Bidding on DoubleClick Ad Exchange

March 19, 2014

25% of all display spending by next yearGoogle Cloud PlatformGoogle Compute EngineAd Exchange trading locationsstrengths of Google Compute EngineGold supporthereOpen Bidderherewhite paperhereAd Exchange Sales.

Tips-and-Tricks: Command-Line Access to Google Cloud Platform using Google Cloud SDK

March 18, 2014

Simplified Tool InstallationGoogle Cloud SDKApp EngineCompute EngineCloud StorageCloud SQLBigQuery

If you’re running on Windows, download Cloud SDK zip and launch install.bat script,

If you’re running on Linux or Mac OS run the following command in your shell/Terminal:

curl https://dl.google.com/dl/cloudsdk/release/install_google_cloud_sdk.bash | bashgcloud.Component Managementgcloud componentscommand groupgcloud components listgcloud components update component-idgcloud components --helpgcloud components update gae-python gae-javaStaying Up-to-Date$ gcloud sql instances list
my-developers-console-project:sql-instance

There are available updates for some Cloud SDK components. To install them, please run:
$ gcloud components updategcloud components updategcloud config set --section component_manager disable_update_checkUnified Authentication into the whole Google Cloud Platformgcloud auth logingcloud auth logingcloud auth listgcloud config set account to set the active account.

Project Initialization and Push-to-Deploy

Every new project created on Google Developers Console now comes with a free private Git repository, where you can store your source code and use it for your App Engine app deployments via push-to-deploy.

To quickly initialize your local environment, run cloud init project-id

. This command will set-up the version control system, clone the source code onto your machine and will set-up push-to-deploy, all in one command. Here’s a workflow example:

Create “my-awesome-app” application in Google Developers Console.

Install Cloud SDK and run gcloud init my-awesome-app.

Change directory to my-awesome-app/default and add some code. (Have a look at some sample projects if you need inspiration.)

Commit the code by running git commit -a -m “First commit for my awesome app” and deploy it by running git push origin master.

Your app should now be serving live at https://my-awesome-app.appspot.com!

General Usage TipsCommand autocompletiongcloudgcloconlgcloud config listgclococonfig componentsgcloud sql instances create Interactive modegcloudgcloud interactivegcloud config listgcloud interactivegcloud.config.list()[’core’][’project’]interactive mode documentation Help and supportgcloud--help

gcloud --help

gcloud sql --help

gcloud sql backups --help

gcloud sql backups list --help

google-cloud-sdk@googlegroups.comdiscussions and support forumgcloud channel at freenode IRC networkgcloud

GA Release for Google APIs Client Library for .NET

March 17, 2014

(cross posted from Google Developers Blog)NuGetOAuth 2.0uploadsdownloadsbatchingfifty Google APIsGetting Startedhere

Compute Engine Management with Puppet, Chef, Salt, and Ansible

March 17, 2014

Google Compute Engine ecosystemCompute Engine Management with Puppet, Chef, Salt, and AnsibleCompute Engine APIgcutilPuppetChefSaltAnsibleCompute Engine Management with Puppet, Chef, Salt, and Ansible

PeopleFun chooses Google App Engine to offload infrastructure tasks and build games people love to play

March 16, 2014

Editor’s note: Today’s guest blog comes from Leon Campise, co-founder of PeopleFun, a Dallas-based creator of social games including Word Chums. Before launching PeopleFun, the company’s founders developed Age of Empires, a highly rated and popular strategy game.
Google App Engine

Improvements for managing Persistent Disks

March 11, 2014

Google Compute EngineCreate and delete a root Persistent Disk with an instanceStarting an Instance in the APISetting the Auto-delete State of a Root Persistent DiskResizing Persistent DisksherehereIncreasing VM Performance To Persistent Disk

Announcing the global Cloud Platform Developer Roadshow

March 11, 2014

Cloud Platform Developer RoadshowGoogle Cloud Platform Liveregister herejoin us

What Lipstick can reveal about your Hadoop pipeline

March 10, 2014

Apache PigNetflix Lipstick

Netflix OSS MeetupNetflix Lipstick on Google Compute Engineresources to help you get started

Announcing updates to the Google Cloud Platform Partner Program

March 6, 2014

Partner Program

Premier Partner - all the benefits of the core partner program and access to premier level services.

Authorized Partner - core partner program with branding, relationship management and access to online resources and training.

Registered Company - entry level status with access to online resources and training.

learn moreapply herepartner directory

Drupal on Google App Engine

March 5, 2014

Today's guest post is from Felipe Rubim, Head of Technology at CI&T. A Top Google Cloud Platform Partner, CI&T builds scalable predictive applications and custom software for fortune 500 companies. As far as PHP and Drupal applications, CI&T has built and maintains more than 400 Drupal enterprise websites globally with an army of 300+ Drupal developers. In this post, Felipe introduces a white paper detailing a test CI&T conducted on Google Cloud Platform and the potential for Drupal Enterprise apps running on Google App Engine PHP. The link to the complete methodology and findings can be found here.Is Google App Engine ready for Drupal and if it is, can it be used for enterprise class applications?Methodology: Findings:

Scalability - We determined that Drupal websites running on top of App Engine scale beautifully. This is especially useful for digital marketing campaigns.

Security - We found App Engine to be in good shape for web portals handling sensitive data.

Versioning - We determined having different versions of your site without going through extra infrastructure set up to be a big game changer.

Drupal Multisite - We concluded this was easy to set up and that it works as expected.

Drush - We found that in order to make Drush work with App Engine, some tweaks are needed when connecting to CloudSQL from another set up (e.g. Google Compute Engine - GCE).

Search - We concluded that installing and configuring Apache Solr in a GCE instance is the way to go for implementing enterprise Drupal search. We created a tutorial for that.

SSO - We also tested SSO with Google Apps and provided a code snippet/tutorial

Is Google App Engine ready for Drupal and if it is, can it be used for enterprise class applications?

Google Drive and Drupal Workflow - leveraging Drupal custom workflows and auditing/logging capabilities vastly improves Google Drive and Drupal Content/Files Workflow/authoring with automatic publishing of approved documents,

App Engine Task Queue - to replace scheduled cron tasks with App Engine Task Queue, and then execute background work (export data from a Drupal user table, campaign/contest registration, etc.) and leverage App Engine interface for management, debugging and logging;

Flexible and extensible stack with an integrated Google Compute Engine approach - The usage of Compute Engine as a complementary component of the solution allows better flexibility to add more features and integration, as well as increased automation for the site management. More details on these features and benefits described below;

Auto-scaleWorry-free infrastructureIntegration with Google AppsApplication ManagemenPotential cost savingsEnterprise to Enterprisehttp://www.ciandt.com/us-en/card/is-google-app-engine-PHP-ready-for-drupal

Snapchat CTO and co-founder Bobby Murphy to appear at Google Cloud Platform Live on March 25

March 4, 2014

SnapchatUrs HölzleGoogle Cloud Platform LiveGoogle Senior Fellow Jeff Deanrequest an invitation to join us in-person in San FranciscoGoogle New York CityGoogle SeattleGoogle Londonyou can always tune in online

Geoip geolocation with Google BigQuery

March 3, 2014

Google BigQueryStack Overflowtransform IP address into geographical locations in BigQuerySELECT COUNT(*) c, city, countryLabel, NTH(1, latitude) lat, NTH(1, longitude) lng
FROM (
SELECT
INTEGER(PARSE_IP(contributor_ip)) AS clientIpNum,
INTEGER(PARSE_IP(contributor_ip)/(256*256)) AS classB
FROM
[publicdata:samples.wikipedia]
WHERE contributor_ip IS NOT NULL
) AS a
JOIN EACH [fh-bigquery:geocode.geolite_city_bq_b2b] AS b
ON a.classB = b.classB
WHERE a.clientIpNum BETWEEN b.startIpNum AND b.endIpNum
AND city != ''
GROUP BY city, countryLabel
ORDER BY 1 DESC

publicdata:samples.wikipediaTechnical detailsGeoLite City IPv4 made available by MaxMindnewer database availableblockslocationsthe new table with the BigQuery web UIcheck out our site"What is BigQuery"Stack Overflowwww.reddit.com/r/bigquery+Google Cloud Platform#BigQueryFelipe HoffaThis post includes GeoLite data created by MaxMind, available from http://www.maxmind.com, distributed under the Creative Commons Attribution-ShareAlike 3.0 Unported License.

Google Cloud Platform Blog