How to create Google Cloud Storage credentials.zip using Spark Shell

This is an example of how to create the credentials.zip fot connecing to Google Cloud Storage.

Open Spark shell for Apache Spark 2.x and Scala 2.11



In [ ]:

    
./bin/spark-shell --packages alvsanand:spark-generic-connector:0.2.0-spark_2x-s_2.11

Import dependencies



In [ ]:

    
import es.alvsanand.sgc.google.GoogleHelper

Create a variable with the JSON of the OAuth 2.0 client ID



In [ ]:

    
val json = """{
  "type": "service_account",
  "project_id": "XXX",
  "private_key_id": "YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY",
  "private_key": "-----BEGIN PRIVATE KEY-----........................-----END PRIVATE KEY-----\n",
  "client_email": "XXX@appspot.gserviceaccount.com",
  "client_id": "ZZZZZZZZZZZZZZZZZZZZZ",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/XXX%40appspot.gserviceaccount.com"
}"""

Create a variable with the destination URL of the credentials zip

Use local path. Note: the zip must be copied to all machines in where Spark Driver/Executors can be executed.



In [ ]:

    
val credentialsZipUrl = "/var/lib/gdc/credentials.zip"

Use HDFS path. Recommended: use this option because the file will be accesible top nay the machines of the Spark cluster.



In [ ]:

    
val credentialsZipUrl = "hdfs://127.0.0.1:8020/var/lib/gdc/credentials.zip"

Execute createCredentialsZip command



In [ ]:

    
val result = GoogleHelper.createCredentialsZip(json, credentialsZipUrl)

Follow instrunctions of the Google API library in the console

   Please open the following address in your browser:
     https://accounts.google.com/o/oauth2/auth?client_id=XXX-XXX.apps.googleusercontent.com&redirect_uri=urn:ietf:wg:oauth:2.0:oob&response_type=code&scope=https://www.googleapis.com/auth/devstorage.read_only%20https://www.googleapis.com/auth/cloud-platform.read-only%20https://www.googleapis.com/auth/devstorage.full_control%20https://www.googleapis.com/auth/cloud-platform%20https://www.googleapis.com/auth/devstorage.read_write
   Attempting to open that address in the default browser now...
   Please enter code: Created new window in existing browser session.

See the result



In [ ]:

    
if (result.isFailute) {
    println("Error creating credentials zip")
}
else {
    println(s"Created credentials zip[$credentialsZipUrl]")
}