Collect to Report with Google Cloud Platform

Collect, store, transform and present all with Google Cloud Platform.

This is my first attempt at Google Cloud Platform (GCP), with a little help on google apps scripts I was able to create this dashboard. I was earlier shy with GCP, thought it needs good coding skills. I am not a coder, but can do some simple scripts with Google search. After dabbling for couple of days in GCP - Apps script, BigQuery and Data studio, Mission Accomplished.

This is not a real time reporting, real time streaming can be done. BigQuery was not required, Google sheet is good enough. Google sheet can store up to 2 Million data cells. If your Google sheet grows, data operations are slower. BigQuery is high performance, primarily for online analytical processing (OLAP) and much more.

My objective was to find ' Where are 'Data Scientist' jobs in US?

This is my approach

1. Collect data from web/API with Google script
2. Save the data to sheet with Google script
3. (Optional) Save data to Bigquery with Google script
4. Report with Data studio (includes data transformation)

There are other better visualization tools, but I found overall 'Collect to Report' process very easy with GCP. Only 300 line of Apps script was used in this project.

(Job API was discontinued in Oct,2020)

To zoom, click zoom icon at bottom right corner 

Google BigQuery, SAP HANA- Cloud Solutions

Week of May 13th, 2013 was a big deal for SAP and Google. Two major events were held-SAPHIRE in Orlando and Developer I/O 2013 in San Francisco. 

I was lucky to get a Google I/O 2013 ticket, and be on of the 6000 participants. My objective was to learn about Google BigQuery and Google Apps Script.

Both companies are competitors in Cloud based 'Analysis or massive database' , SAP HANA has two product versions 'Appliance ' (Hardware and Software) and 'Cloud'. Where as Google has Cloud solution BigQuery. 

SWOT	SAP HANA Appliance	SAP HANA Cloud	Google BigQuery
Strengths	- Strong SAP Products/Branding	- No downtime - Good for Ad hoc requirements	- Good Enterprise Cloud offerings - No downtime - Good for Ad hoc requirements - Good Developer community
Weakness	- Maybe costs and downtime	- Dependency on AWA (Amazon web services) for cloud storage	- Provides tools, not business suits - some products are evolving (e.g. Data Visualization)
Opportunities	- Increase production adoption - Conversion from Cloud version - Use as database for other applications	- Use as database for other applications - Convert clients with web security concerns - Encourage Developer community to build solutions	- Use as database for other applications (using Compute Engine) - Convert clients with web security concerns
Threats	- Cloud products, and other Appliances	- Google BigQuery, Maybe Amazon	- Maybe Amazon

There were two client presentations on BigQuery - Gamesys and Shutterfly, these clients also use Google Apps Script for data visualization/reporting.

Here are the two Google BigQuery recordings
Big Data Mashups: Enabling Next Generation Analytics Using BigQuery
Here Be BigQuery: Building Social Gaming Infrastructure on the Google Cloud Platform

Another interesting project was All the Ships in the World: Visualizing Data with Google Cloud and Maps, which integrate many Google technologies to visualize real time Data.

I see SAP ideologically now being more open, proving more resources to developers and planning more Cloud based products.

It will be interesting to see how enterprise Cloud database and analysis tools will evolve and their acceptance.