googlecloudplatform-test: Google BigQuery adds UDF support for deeper cloud analytics

Google Cloud Platform Blog

Product updates, customer stories, and tips and tricks on Google Cloud Platform

Google BigQuery adds UDF support for deeper cloud analytics

August 25, 2015

We recently announced that Google Cloud Dataflow and Google Cloud Pub/Sub graduated to general availability. You can now leverage these easy to use and inexpensive large-scale fully managed big data services with Google BigQuery to find valuable business information and insights.

BigQuery is a No-Ops analytics database that seamlessly scales in seconds, requires no instance or cluster management, offers unbeatable performance out-of-the-box, and lets you pay only for what you consume. Today, we’re releasing a new version of BigQuery that is easier to use, more powerful and more open.

With new features such as User-Defined Functions (UDFs) and an improved user interface (UI), BigQuery is now simpler and easier to use.

User-Defined Functions (UDFs). Expressed in Javascript, UDFs allow you to extend SQL and execute arbitrary code within BigQuery. For example, you can now easily express complex conditional logic in your queries and you have much more flexibility than provided by regular expressions. Head over to our documentation to learn more.

Query files in Google Cloud Storage from BigQuery. It is now possible to run queries without loading files into BigQuery first. This functionality also simplifies data import into BigQuery. In addition to the existing straight “import” mechanism, you can now write queries which read from Cloud Storage files and write the results to BigQuery tables. Federated query documentation offers more details.

Increased query limits. You will now be able to run 50 simultaneous queries, and 100,000 queries per day (up from 20 and 20,000). In addition, there will no longer be limits on “maximum simultaneous bytes processed” and “maximum simultaneous large queries”. These changes give you more freedom within the BigQuery ecosystem.

UI Improvements. We’ve added several new features, including a new “Format Query” button, automatic organization of date-sharded tables, and the ability to download query results in JSON.

We also wanted to make BigQuery more powerful and performant to help you save time and increase productivity.

Dynamic query optimization. Improves reliability and performance for complex queries such as large JOIN or GROUP BY operations. You can expect to see your project activated in the coming weeks. Users will no longer need to specify the EACH keyword, which greatly simplifies the writing of queries, particularly for applications that programmatically generate SQL such as visualization tools and dashboards.

Enhancements to the query execution engine will result in increased performance and scale of queries that use lots of resources, such as large JOINs, analytic functions, and high-cardinality aggregations.

And lastly, we added new features to BigQuery to be more open.

BigQuery Slots. One unique feature of BigQuery is the ability to dip into the vast shared pool of resources to scale into thousands of cores for a query. BigQuery Slots offer customers the ability to expand and allot the resources available to them, regardless of system load. Use cases include latency-sensitive SaaS, ETL, and business reporting workloads.

High-Compute Pricing Tiers. With release of UDFs, Dynamic query optimization, and execution engine improvements, BigQuery now supports queries that consume large amounts of compute resources relative to “bytes scanned”. To enable this higher resource consumption, we are introducing High Compute Pricing Tiers. For more information, head over to our pricing page.

BigQuery is fully-managed by Google, so customers automatically get all these benefits right away. Make use of the of better UI, better performance, and additional functionalities - no action needed, and no downtime. Solve your big data problems the way we solve ours!

Learn how BigQuery can help you, take a look at the documentation, and try it out! First terabyte processed is on us!

- Posted by Tino Tereshko, Technical Program Manager