

In [0]:

    
#@title Copyright 2020 Google LLC. { display-mode: "form" }
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

View source on GitHub

Uploading an image from tiles using a manifest

This notebook demonstrates uploading a set of image tiles into a single asset using a manifest file. See this doc for more details about manifest upload using the Earth Engine command line tool.

10-meter land cover images derived from Sentinel-2 (reference) from the Finer Resolution Global Land Cover Mapping (FROM-GLC) website are downloaded directly to a Cloud Storage bucket and uploaded to a single Earth Engine asset from there. A manifest file, described below, is used to configure the upload.

First, authenticate with Google Cloud, so you can access Cloud Storage buckets.



In [0]:

    
from google.colab import auth
auth.authenticate_user()

Download to Cloud Storage

Paths from the provider website are manually copied to a list object as demonstrated below. Download directly to a Cloud Storage bucket to which you can write.



In [0]:

    
# URLs of a few tiles.
urls = [
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_36_-120.tif',
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_36_-122.tif',
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_36_-124.tif',
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_38_-120.tif',
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_38_-122.tif',
  'http://data.ess.tsinghua.edu.cn/data/fromglc10_2017v01/fromglc10v01_38_-124.tif'
]

# You need to have write access to this bucket.
bucket = 'your-bucket-folder'

# Pipe curl output to gsutil.
for f in urls:
  filepath = bucket + '/' + f.split('/')[-1]
  !curl {f} | gsutil cp - {filepath}

Build the manifest file

Build the manifest file from a dictionary. Turn the dictionary into JSON. Note the use of the gsutil tool to get a listing of files in a Cloud Storage bucket (learn more about gsutil). Also note that the structure of the manifest is described in detail here. Because the data are categorical, a MODE pyramiding policy is specified. Learn more about how Earth Engine builds image pyramids here.



In [0]:

    
# List the contents of the cloud folder.
cloud_files = !gsutil ls {bucket + '/*.tif'}

# Get the list of source URIs from the gsutil output.
sources_uris = [{'uris': [f]} for f in cloud_files]

asset_name = 'path/to/your/asset'

# The enclosing object for the asset.
asset = {
  'name': asset_name,
  'tilesets': [
    {
      'sources': sources_uris
    }
  ],
  'bands': [
    {
      'id': 'cover_code',
      'pyramiding_policy': 'MODE',
      'missing_data': {
          'values': [0]
      }
    }
  ]
}

import json
print(json.dumps(asset, indent=2))

Inspect the printed JSON for errors. If the JSON is acceptable, write it to a file and ensure that the file matches the printed JSON.



In [0]:

    
file_name = 'gaia_manifest.json'

with open(file_name, 'w') as f:
  json.dump(asset, f, indent=2)

Inspect the written file for errors.



In [0]:

    
!cat {file_name}

Upload to Earth Engine

If you are able to cat the written file, run the upload to Earth Engine. First, import the Earth Engine library, authenticate and initialize.



In [0]:

    
import ee
ee.Authenticate()
ee.Initialize()



In [0]:

    
# Do the upload.
!earthengine upload image --manifest {file_name}

Visualize the uploaded image with folium

This is what FROM-GLC says about the classification system:

Class	Code
Cropland	10
Forest	20
Grassland	30
Shrubland	40
Wetland	50
Water	60
Tundra	70
Impervious	80
Bareland	90
Snow/Ice	100

Use a modified FROM-GLC palette to visualize the results.



In [0]:

    
palette = [
    'a3ff73', # farmland
    '267300', # forest
    'ffff00', # grassland
    '70a800', # shrub
    '00ffff', # wetland
    '005cff', # water
    '004600', # tundra
    'c500ff', # impervious
    'ffaa00', # bare
    'd1d1d1', # snow, ice
]
vis = {'min': 10, 'max': 100, 'palette': palette}

ingested_image = ee.Image('projects/ee-nclinton/assets/fromglc10_demo')
map_id = ingested_image.getMapId(vis)

import folium

map = folium.Map(location=[37.6413, -122.2582])
folium.TileLayer(
  tiles=map_id['tile_fetcher'].url_format,
  attr='Map Data &copy; <a href="https://earthengine.google.com/">Google Earth Engine</a>',
  overlay=True,
  name='fromglc10_demo',
).add_to(map)
map.add_child(folium.LayerControl())
map