Building Integrated Applications on Google's Cloud Technologies

#HELLO #BERLIN
I'm Michael Manoochehri

• Developer Relations Engineer: Cloud and Apps
• Working with BigQuery Team
• I <3 Big Data
Find me on all the social networks:
• Google+, Twitter (@nTangledMichael), etc
• #clo3 #gddde

Agenda

• App Engine Recap
• Google's new Cloud Technologies
• Google Storage
• Prediction API
• We'll build a simple Prediction Web App
• BigQuery
• Summary and Q&A

Timeline of Google's Search Index

• 1998: Google 'founded' - index contained 26 million pages
• 2000: Google's index reached the one billion mark
• 2004: Google's index contained 4.28 billion web pages and 880 million images
• 2008: Index reached 1 trillion unique URLs

Cloud Development in a Box

• Downloadable SDK
• Various application runtimes: Java, Python, and Go
• Local development tools: Eclipse plugin, GAE launcher
• Specialized application services: memcache, xmpp, mail, backends, etc
• 15:30 - Accelerated Beginner App Engine Workshop in this room

Google Cloud Storage for Developers: What is it?

• Store your data in Google's cloud
• Massive file sizes... store terabytes
• You control access to your data with ACLs
• Private, shared, or public
• Access via Google APIs or 3rd party tools/libraries

Google Cloud Storage: Tech Summary

• Built for Performance, Scale, and Consistency
• Consistency and Data Replication:
• All data replicated to multiple US data centers
• Leverages Google's worldwide network for data delivery
• Can request resumable uploads
• Can request US or EU datacenter location

Google Cloud Storage for Developers: Use Cases

• Static content hosting
• Data storage for applications
• Backup and recovery
• Storage for Computation

Google Cloud Storage within Google

Google Cloud Storage: Early Adopters

Google Cloud Storage: Tools Demo!

• gsutil: command line storage tool
• Google Storage Manager: Web UI interface

Google Cloud Storage: Summary

• Uses Google's infrastructure
• Excellent security, permissions and great uptime
• Lots of tools and third party libraries
• S3 compatible
• Use it with App Engine!

Cloud Storage as a Platform for Google's Big Data Products

What is the Prediction API?

• Categorical Classification
• Is this email SPAM?
• Is this phrase in Spanish or French?
• Regression values
• Provide an expected numerical value based on various features
• What will the expected temperature be based on a collection of values?

Prediction API finds relevant features in the sample data during training.

Prediction API later searches for those features during prediction.

Using the Prediction API: 3 steps

1. Upload your training data to Google Storage
2. Build a model from your data
3. Make new predictions!

Step 1: Upload your training data to Google Storage

• Training data: outputs and input features
• Data format: comma separated value format (CSV)

"english","To err is human, but to really ..."
"french","Le plus grand faible des hommes, c'est l'amour..."
	

• Upload to Google Storage

gsutil cp ${data} gs://yourbucket/${data}
    

Step 2: Train the Prediction API with your data model

• To train a model:

POST prediction/v1.4/training
{"id":"mybucket/mydata"}
	

sh training.sh [model_name]
    

Step 2a: Train the Prediction API with your data model

• Training runs asynchronously. To see if it has finished:

GET prediction/v1.4/training/mybucket%2Fmydata

{"kind": "prediction#training",...
  ,"training status": "DONE"}
	

sh check-training.sh [model_name]
    

Step 3: Apply the trained model to make predictions on new data

POST prediction/v1.4/training/mybucket%2Fmydata/predict
  { "data":{ "input": { "text" : [
    "J'aime X! C'est le meilleur" ]}}}    

sh predict.sh [model_name] "'Man is still the most \
                            extraordinary computer of all'"

Prediction API: Best Practices

• Categorical predictions: Try to have hundreds of examples per category - a few dozen at minimum
• Big data tends to beat a smarter algorithm
• 10 times as many examples as features
• You have 4 different features? You need at least 40 examples!
• The more unique features, the better!
• For text categorization, the size of model features should match size of query text
• Compare tweets to tweet, not tweets to Shakespeare

What is BigQuery?

• Google's large data adhoc analysis technology
• Yes, we use this internally, and even wrote a few papers about it!
• Analyze massive amounts of data in seconds
• Simple SQL-like query language
• REST APIs, JSON-RPC, Google Apps Script

BigQuery is designed to excel at 2 things...

• Scale: Billions of rows! Terabytes of data!
• Query speed: Seconds instead of minutes... or hours...

How does BigQuery Work?

What can you do with BigQuery?

• Create dashboards and reporting tools
• Find patterns in large datasets
• Create new analytics products
• Business analysis tools
• Lots of things we haven't thought of yet

• This Presentation:
• http://goo.gl/StlK0
• http://gdd-2011-cloud-presentation.appspot.com/
• Connect with me:
• Michael Manoochehri
• Google+: gplus.to/manoochehri
• Twitter: @nTangledMichael
• etc...