notebook.community

Edit and run

s3

list all files





In [15]:



    


import boto3
s3=boto3.client('s3')
list=s3.list_objects(Bucket='mert01')['Contents']
list[1:3]



    


















    
Out[15]:








[{'ETag': '"8e6fa0797999fc54bedc8d18a2b0b3d6"',
  'Key': '2013-02-23-12-17-49-6B4C46A82B207DC2',
  'LastModified': datetime.datetime(2013, 2, 23, 12, 17, 50, tzinfo=tzutc()),
  'Owner': {'DisplayName': 's3-log-service',
   'ID': '3272ee65a908a7677109fedda345db8d9554ba26398b2ca10581de88777e2b61'},
  'Size': 366,
  'StorageClass': 'STANDARD'},
 {'ETag': '"32637e53da6c0afc8ce12a9b84559106"',
  'Key': '2013-02-23-13-18-48-629C73E5A218D37F',
  'LastModified': datetime.datetime(2013, 2, 23, 13, 18, 49, tzinfo=tzutc()),
  'Owner': {'DisplayName': 's3-log-service',
   'ID': '3272ee65a908a7677109fedda345db8d9554ba26398b2ca10581de88777e2b61'},
  'Size': 367,
  'StorageClass': 'STANDARD'}]

















In [27]:



    


# https://stackoverflow.com/questions/3337912/quick-way-to-list-all-files-in-amazon-s3-bucket
import boto

s3 = boto.connect_s3()
bucket = s3.get_bucket('mert01')
#bl = bucket.list()
#for key in bucket.list():
#    print(key.name)



    











In [30]:



    


import itertools
for el in itertools.islice(bl, 0, 3):
    print(el.name)



    


















    






2013-02-23-11-18-40-F481822430A3900B
2013-02-23-12-17-49-6B4C46A82B207DC2
2013-02-23-13-18-48-629C73E5A218D37F

















In [32]:



    


# len(bl)
i = 0
for key in bucket.list():
    i = i + 1
print(i)



    


















    






23949

















In [37]:



    


bl2 = bucket.list(prefix="201")
for el in itertools.islice(bl2, 0, 3):
    print(el.name)



    


















    






2013-02-23-11-18-40-F481822430A3900B
2013-02-23-12-17-49-6B4C46A82B207DC2
2013-02-23-13-18-48-629C73E5A218D37F

















In [38]:



    


# https://stackoverflow.com/questions/10054985/how-to-delete-files-recursively-from-an-s3-bucket#18698235
result = bucket.delete_keys([key.name for key in bl2])
result



    


















    
Out[38]:








<boto.s3.multidelete.MultiDeleteResult at 0x107cf8978>

aws cli

aws configure





In [ ]:



    


!aws configure help



    


















    






CONFIGURE()                                                        CONFIGURE()



NAME
       configure -

DESCRIPTION
       Configure  AWS  CLI  options. If this command is run with no arguments,
       you will be prompted for configuration values such as your  AWS  Access
       Key  Id  and you AWS Secret Access Key.  You can configure a named pro-
       file using the --profile argument.  If your config file does not  exist
       (the default location is ~/.aws/config), the AWS CLI will create it for
       you.  To keep an existing value, hit enter when prompted for the value.
       When  you  are prompted for information, the current value will be dis-
       played in [brackets].  If the config item has no value, it be displayed
       as  [None].  Note that the configure command only work with values from
       the config file.  It does not use any configuration values  from  envi-
       ronment variables or the IAM role.

       Note:  the  values  you  provide  for the AWS Access Key ID and the AWS
       Secret Access Key will  be  written  to  the  shared  credentials  file
       (~/.aws/credentials).
:
















In [2]:



    


!aws configure list



    


















    






      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ****************YBAQ shared-credentials-file    
secret_key     ****************91un shared-credentials-file    
    region             eu-central-1      config-file    ~/.aws/config

















In [3]:



    


!cat ~/.aws/config



    


















    






[default]
region = eu-central-1
output = json

















In [5]:



    


!export AWS_DEFAULT_REGION=us-west-1
!aws ec2 describe-volumes



    


















    






{
    "Volumes": []
}

IAM (identity and access management)

Documentation:

http://docs.aws.amazon.com/IAM/latest/UserGuide/getting-started_create-admin-group.html?shortFooter=true#Using_CreateGroup





In [6]:



    


!aws iam list-groups



    


















    






{
    "Groups": [
        {
            "Path": "/",
            "CreateDate": "2017-03-27T12:19:15Z",
            "Arn": "arn:aws:iam::217382780035:group/iterative",
            "GroupName": "iterative",
            "GroupId": "AGPAJEEYNNP5NEHOFFSHO"
        }
    ]
}

















In [7]:



    


!aws iam list-attached-group-policies --group-name iterative



    


















    






{
    "AttachedPolicies": [
        {
            "PolicyName": "AdministratorAccess",
            "PolicyArn": "arn:aws:iam::aws:policy/AdministratorAccess"
        }
    ]
}

















In [1]:



    


!aws iam list-users



    


















    






{
    "Users": [
        {
            "UserId": "AIDAI3KD6Q7I5CKXBS2B2",
            "UserName": "Administrator",
            "PasswordLastUsed": "2017-03-27T12:24:20Z",
            "Path": "/",
            "CreateDate": "2017-03-27T12:19:48Z",
            "Arn": "arn:aws:iam::217382780035:user/Administrator"
        },
        {
            "UserId": "AIDAIUJA3O4KRIVFN3QU6",
            "UserName": "mertnuhoglu",
            "Path": "/",
            "CreateDate": "2012-02-07T13:15:07Z",
            "Arn": "arn:aws:iam::217382780035:user/mertnuhoglu"
        }
    ]
}

Find cheapest prices: awespottr





In [3]:



    


!awespottr c4.xlarge



    


















    






Checking spot prices for c4.xlarge instance type.

AWS Zone                 Hourly Rate 
------------------------ ------------
us-east-2b               $0.025200   
us-east-2c               $0.025400   
us-east-2a               $0.032000   
ap-northeast-2a          $0.033500   
ap-northeast-2c          $0.034000   
ca-central-1b            $0.035000   
eu-west-2b               $0.035000   
eu-west-2a               $0.038000   
us-west-1a               $0.038000   
us-west-1b               $0.039400   
ap-southeast-1b          $0.040700   
sa-east-1c               $0.048000   
eu-central-1b            $0.048100   
us-east-1f               $0.048200   
us-east-1a               $0.048500   
us-east-1d               $0.049000   
us-east-1b               $0.049100   
us-east-1c               $0.049900   
us-east-1e               $0.050100   
ap-south-1a              $0.051700   
ca-central-1a            $0.053400   
us-west-2b               $0.057400   
us-west-2a               $0.058000   
us-west-2c               $0.058000   
ap-south-1b              $0.060000   
ap-northeast-1a          $0.064600   
ap-northeast-1c          $0.064700   
sa-east-1a               $0.066800   
ap-southeast-2b          $0.070800   
ap-southeast-2c          $0.070900   
ap-southeast-2a          $0.071200   
eu-west-1c               $0.072200   
eu-west-1b               $0.072300   
eu-west-1a               $0.073200   
eu-central-1a            $0.073500   
ap-southeast-1a          $0.078400   
eu-central-1c            $0.091100   
ap-northeast-1b          $2.940000   
sa-east-1b               $3.250000   

Cheapest hourly rate for c4.xlarge is $0.025200 in zone us-east-2b

















In [ ]:



    


!export AWS_DEFAULT_REGION=us-east-2
!aws ec2 describe-spot-price-history --availability-zone "${AWS_DEFAULT_REGION}b" --product-description "Linux/UNIX" --instance-types c4.xlarge --start-time `date -u --date="7 days ago" +'%Y-%m-%dT%H:%M:00'` | jq -r -c '.SpotPriceHistory[] | (.SpotPrice)' | head -n 20



    











In [1]:



    


!aws ec2 describe-regions



    


















    






{
    "Regions": [
        {
            "Endpoint": "ec2.ap-south-1.amazonaws.com",
            "RegionName": "ap-south-1"
        },
        {
            "Endpoint": "ec2.eu-west-2.amazonaws.com",
            "RegionName": "eu-west-2"
        },
        {
            "Endpoint": "ec2.eu-west-1.amazonaws.com",
            "RegionName": "eu-west-1"
        },
        {
            "Endpoint": "ec2.ap-northeast-2.amazonaws.com",
            "RegionName": "ap-northeast-2"
        },
        {
            "Endpoint": "ec2.ap-northeast-1.amazonaws.com",
            "RegionName": "ap-northeast-1"
        },
        {
            "Endpoint": "ec2.sa-east-1.amazonaws.com",
            "RegionName": "sa-east-1"
        },
        {
            "Endpoint": "ec2.ca-central-1.amazonaws.com",
            "RegionName": "ca-central-1"
        },
        {
            "Endpoint": "ec2.ap-southeast-1.amazonaws.com",
            "RegionName": "ap-southeast-1"
        },
        {
            "Endpoint": "ec2.ap-southeast-2.amazonaws.com",
            "RegionName": "ap-southeast-2"
        },
        {
            "Endpoint": "ec2.eu-central-1.amazonaws.com",
            "RegionName": "eu-central-1"
        },
        {
            "Endpoint": "ec2.us-east-1.amazonaws.com",
            "RegionName": "us-east-1"
        },
        {
            "Endpoint": "ec2.us-east-2.amazonaws.com",
            "RegionName": "us-east-2"
        },
        {
            "Endpoint": "ec2.us-west-1.amazonaws.com",
            "RegionName": "us-west-1"
        },
        {
            "Endpoint": "ec2.us-west-2.amazonaws.com",
            "RegionName": "us-west-2"
        }
    ]
}

















In [2]:



    


!aws ec2 describe-availability-zones --region us-east-2



    


















    






{
    "AvailabilityZones": [
        {
            "ZoneName": "us-east-2a",
            "Messages": [],
            "State": "available",
            "RegionName": "us-east-2"
        },
        {
            "ZoneName": "us-east-2b",
            "Messages": [],
            "State": "available",
            "RegionName": "us-east-2"
        },
        {
            "ZoneName": "us-east-2c",
            "Messages": [],
            "State": "available",
            "RegionName": "us-east-2"
        }
    ]
}

Finding AMIs





In [3]:



    


!aws ec2 describe-images --owners self amazon --filters "Name=root-device-type,Values=ebs" > data/ex_aws01.json



    











In [4]:



    


!cat data/ex_aws01.json | head -n 30



    


















    






{
    "Images": [
        {
            "ImageId": "ami-0083a365",
            "Platform": "windows",
            "RootDeviceType": "ebs",
            "VirtualizationType": "hvm",
            "Hypervisor": "xen",
            "CreationDate": "2017-08-12T07:45:04.000Z",
            "Description": "Microsoft Windows Server 2008 SP2 Datacenter 64-bit Locale English with SQL Express 2008 AMI provided by Amazon",
            "ImageOwnerAlias": "amazon",
            "ImageType": "machine",
            "Architecture": "x86_64",
            "Public": true,
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "SnapshotId": "snap-0681d17022194981a",
                        "DeleteOnTermination": true,
                        "VolumeType": "gp2",
                        "Encrypted": false,
                        "VolumeSize": 50
                    }
                },
                {
                    "DeviceName": "xvdca",
                    "VirtualName": "ephemeral0"
                },
                {
cat: write error: Broken pipe

















In [2]:



    


!jq '{ami: .Images[].ImageId}' data/ex_aws01.json | head -n 8



    


















    






{
  "ami": "ami-0083a365"
}
{
  "ami": "ami-009cbc65"
}
{
  "ami": "ami-00e9cb65"

















In [1]:



    


!jq '{ami: [.Images[].ImageId]}' data/ex_aws01.json | head -n 5



    


















    






{
  "ami": [
    "ami-0083a365",
    "ami-009cbc65",
    "ami-00e9cb65",

















In [ ]:

Content source: mertnuhoglu/study

Similar notebooks:

notebook.community | gallery | about