(Run the last cell first in order to enable custom formatting)

Learning Pandas

AMCDawes

Dec 2015

Some parts of our CCDimage code would be much improved by the use of pandas. In particular, some statistics and other anaylsis would be very straightforward. This is a place for my notes as I learn pandas.


In [38]:
# standard imports:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib

# use inline plots:
%matplotlib inline

# use ggplot style:
matplotlib.style.use('ggplot')

Our data is currently represented as a stack of 2D arrays (so a 3D "array"). In that implementation the third index is essentially a shot number and the first two are pixel row and pixel column. We'd like to make a dataframe of this data and then work with it in pandas instead of in an array. Another data type we have is the $K_p$ values (essentially the FFT output). In this case, it is a 1D array for each shot of data. The pandas version would be a dataframe with each element being a 1D array of $K_p$ values.

To test this, let's create a random dummy set that is 20 shots of 400 $K_p$ values:


In [10]:
# create an example dataframe:
df = pd.DataFrame(np.random.randn(20,400), index=np.arange(20), columns=1.23*np.arange(400))

In [11]:
df


Out[11]:
0.0 1.23 2.46 3.69 4.92 6.15 7.38 8.61 9.84 11.07 ... 479.7 480.93 482.16 483.39 484.62 485.85 487.08 488.31 489.54 490.77
0 0.184962 -0.593215 0.189440 -0.041520 -0.468275 1.248782 -1.325620 0.712231 -0.162094 1.817380 ... -1.343064 -1.470482 0.735458 0.679417 -0.240810 -0.455247 -0.493477 0.546133 -0.093801 -0.870588
1 0.330364 1.285958 0.221395 -0.679064 -0.605855 -1.138653 0.432561 -0.072273 -0.031589 -0.955335 ... 0.403545 -0.294596 -0.046364 1.550566 1.322492 0.158821 0.156565 -0.445687 -2.098845 -1.161122
2 -0.248033 -0.075215 1.170656 0.055841 -0.277661 0.005537 0.479036 0.754289 -0.441990 -1.282790 ... -0.323793 -3.010945 0.336506 -0.325221 0.376113 0.062601 -0.838108 -1.102468 0.338594 0.423082
3 1.721958 -0.023670 -1.054532 1.613846 0.368450 1.702158 -0.076590 -1.169031 0.483357 0.935780 ... -0.178595 1.032906 -2.016839 2.064577 1.685427 -0.653712 -1.918819 0.239802 0.550428 -1.531779
4 1.571422 -0.108286 0.329186 -0.399168 -0.406936 0.530031 -1.320046 -1.747431 -1.522011 1.004620 ... -1.139332 0.549895 1.571382 0.940728 -0.602542 0.138620 -0.593473 0.266806 -0.832163 -1.852263
5 -0.086009 0.595296 0.425701 1.522184 1.367878 1.281471 -0.541161 1.061773 -0.184506 -0.372571 ... 0.419337 -0.072765 2.219141 -0.960565 -1.141693 0.024791 -1.159445 1.722109 -0.048243 1.181456
6 0.663104 1.017888 -0.049473 -0.651722 0.167377 0.377671 -0.467855 0.071341 0.123341 -0.587914 ... -1.230849 -1.044841 0.162082 -0.314299 -0.799146 -0.199157 0.314127 0.213541 1.427636 0.283838
7 -0.419924 -0.354931 0.916886 -0.751909 1.013355 -1.736379 0.522979 0.470240 -0.634015 -0.360114 ... 1.375368 -0.968433 0.362897 -1.411900 0.530603 1.166181 0.625223 -0.266564 -0.570224 -0.822201
8 0.168113 0.887247 -1.040347 0.194861 0.442004 0.275889 -0.243377 1.089206 0.664697 0.223828 ... -0.928463 0.471350 0.490348 0.217929 -0.883274 -0.421726 1.486858 0.689218 0.754216 0.110106
9 1.186498 -0.960690 -0.348566 0.990295 0.675831 0.761218 -0.542745 0.261024 0.001028 1.173646 ... -0.059401 0.616330 -1.674507 -1.064985 0.535332 -1.322795 -1.539502 0.532675 0.173978 1.202523
10 -1.352537 -0.425472 0.879962 -0.239729 -0.233018 0.734855 1.006921 0.607420 -1.669254 -0.693078 ... -0.659705 0.788002 0.248544 -1.297290 0.316705 -0.207468 -0.405590 0.842259 0.514191 0.423534
11 1.599110 -0.100259 0.878009 -0.890536 1.438008 1.520079 -1.695664 -1.685915 -0.833511 0.277582 ... 1.975432 0.831157 -0.004321 -0.091759 -0.384815 -1.432872 1.437886 2.025241 -1.649293 -1.123986
12 0.663158 -0.613900 0.652613 1.288169 0.154322 1.274050 0.246589 0.623908 -0.535676 0.181517 ... -1.122317 -2.191370 -0.585120 -1.325936 -0.674683 -1.193839 0.398804 0.260643 -0.042832 -0.139533
13 -0.645737 -0.510152 -1.005171 -1.624530 0.898106 0.066679 0.874028 -1.401223 -0.057903 0.582753 ... -1.380020 -2.512605 1.118588 0.634844 0.301920 0.236669 -0.509022 0.171967 -0.814506 -0.306371
14 1.321814 -0.774301 -1.902708 1.957949 2.112021 0.815829 -2.043558 -0.020623 0.595277 -0.107277 ... -0.427886 -0.090893 0.283438 0.656301 0.766152 -1.237433 1.099856 -0.036330 -1.391981 0.906868
15 -0.803413 0.036083 -0.761519 -1.115538 0.178751 -0.937486 0.237419 0.474481 -1.090644 0.645634 ... -1.528687 0.021469 0.147363 0.549197 0.077641 -1.080116 1.058282 0.961617 0.005075 0.570475
16 0.761742 0.049620 0.740389 -1.541954 -0.477572 -0.396389 0.217273 -0.255407 0.067497 0.006614 ... 0.082712 0.449123 -0.875030 -0.495502 -2.268136 0.375531 -1.265048 0.530770 -1.081555 -1.910912
17 1.494090 0.608071 -0.028324 0.413887 0.340919 -0.401112 -0.439394 -0.441322 -0.155351 1.351053 ... -1.021816 -0.639324 0.622632 0.691260 -0.438366 0.321922 -0.595063 1.005989 -0.511210 -0.248449
18 1.839444 0.250622 0.030625 0.658200 -0.525802 1.183947 0.920444 0.297210 0.194203 0.366299 ... 1.459953 0.212683 0.216094 -0.430931 -0.885898 -0.151657 -0.829574 0.051659 0.822611 -0.310205
19 -1.863444 1.019783 0.423566 -1.553650 0.802445 0.083260 -0.491350 2.118290 0.345122 2.467606 ... 0.481893 1.324027 -0.285335 -1.192188 1.645913 -0.321261 2.514560 -0.868827 -0.813402 -0.029505

20 rows × 400 columns

So we see a few nice features:

  • we can label the columns with useful things, this could be the actual mode index $p$ for the columns
  • we can also label rows, this could be shot number, or even a alphanumeric code for shot/run (can it be alphanumeric?)

To show this, we'll create a list of the row labels using a list comprehension. Note, this could also be done with a generator, but that is a more advanced pythonism


In [12]:
# nested string list comprehension:
rows = ["r{}s{}".format(i,j) for i in np.arange(4) for j in np.arange(5)]

In [13]:
# could also be a generator:
row_gen = ("r{}s{}".format(i,j) for i in np.arange(4) for j in np.arange(5))

In [14]:
for i in row_gen:
    print(i)


r0s0
r0s1
r0s2
r0s3
r0s4
r1s0
r1s1
r1s2
r1s3
r1s4
r2s0
r2s1
r2s2
r2s3
r2s4
r3s0
r3s1
r3s2
r3s3
r3s4

In [15]:
df2 = pd.DataFrame(np.random.randn(20,400), index=rows, columns=1.23*np.arange(400))

In [16]:
df2


Out[16]:
0.0 1.23 2.46 3.69 4.92 6.15 7.38 8.61 9.84 11.07 ... 479.7 480.93 482.16 483.39 484.62 485.85 487.08 488.31 489.54 490.77
r0s0 -0.075841 -0.904605 -1.094072 0.605669 -1.935258 0.650063 0.660024 -2.170779 1.076856 -0.521928 ... 1.028115 -0.131176 1.219894 0.364914 1.028008 -0.210408 0.584433 0.022389 0.198800 2.394047
r0s1 0.464307 0.507541 -0.215419 -0.744910 1.049569 -0.146941 0.288519 0.591788 -1.127253 -0.838182 ... 0.611890 -0.847824 -0.776338 -0.056745 0.168138 -0.413380 -0.136623 1.508007 0.763493 -1.961929
r0s2 0.549017 -0.010613 0.350341 0.600244 0.564113 -0.135829 -0.530846 -0.960254 0.759039 -0.449446 ... 0.624111 -0.088894 -1.126249 0.604857 -1.559714 1.838663 -1.085178 -1.633170 -0.223984 -0.287379
r0s3 0.854825 1.083141 -1.502872 0.215501 1.037609 0.905728 -0.099644 -0.872863 0.409028 -0.392250 ... -1.517258 0.598385 -0.585482 0.333990 -0.100666 -0.024075 -0.180373 -1.391312 2.034463 1.557063
r0s4 -0.540168 -2.024637 1.521336 0.458756 0.031326 -0.362836 0.002126 -0.623581 0.187056 -1.338258 ... 0.124124 1.468026 0.705380 -1.086340 -1.358901 -0.294727 -0.324702 -0.815423 0.256468 1.090099
r1s0 0.826986 -0.842467 1.049514 1.150837 -0.785820 0.429018 1.115340 1.307672 0.376442 0.854339 ... -0.429645 -0.675573 -0.810689 0.187499 0.262096 0.496407 -0.978998 -0.594335 -0.133461 2.547492
r1s1 -2.407435 0.084073 0.926649 -1.474173 1.498178 0.161716 1.031246 -0.094132 -0.994738 -1.276429 ... 0.244866 0.988604 -0.210704 -0.633346 -1.639042 2.043875 0.916145 1.058196 -0.923993 0.696969
r1s2 -0.553663 0.260108 0.565609 0.348731 -0.222907 1.454296 -0.804949 -1.058964 0.146190 1.406294 ... 0.778546 -0.583378 -1.960605 -2.030100 0.323722 -1.504652 -0.222541 0.062490 0.261073 0.655042
r1s3 -1.923620 1.897037 1.345015 -1.532653 0.272023 0.384065 1.002208 0.272557 -0.646553 -0.312436 ... -0.196265 1.047866 0.542254 0.372460 0.896880 0.828275 -1.182366 -0.272894 -0.625088 0.087952
r1s4 -1.186381 0.669413 0.760152 -0.552006 0.521368 -0.949405 -0.020145 -0.090695 -1.702185 -0.757083 ... 1.690140 0.634193 -1.026187 1.614171 -0.463188 0.336056 -1.407198 -1.658494 -2.787294 -0.790165
r2s0 0.990522 -0.961763 0.439112 0.885652 1.450123 -0.499534 0.938854 1.408904 -0.740930 -0.140112 ... -0.063761 -0.626640 0.041427 -0.345510 0.431423 -0.716702 1.691901 -0.428716 -0.836942 -0.288951
r2s1 -0.519180 -1.298858 0.250681 -1.190024 0.106520 -0.095677 0.615137 0.495602 -0.212434 -0.349042 ... -1.641367 -0.665836 0.256332 0.451142 1.305498 -0.108231 0.204788 0.869855 -0.266288 1.509643
r2s2 0.734422 0.476790 -0.458226 -0.700024 -1.238566 0.599671 -0.427914 0.844219 -0.270551 0.492769 ... -1.239655 1.126015 -0.550496 -0.761821 -0.531756 -0.332588 -0.523914 -0.290804 0.246082 -0.102039
r2s3 -0.853220 -2.093568 -0.643340 0.200368 -0.716152 -1.126140 -0.698815 -0.416204 -0.544038 0.368496 ... -1.466966 1.246151 0.064958 -1.270758 1.034440 -0.975365 -0.009970 -0.033534 -1.204973 1.992776
r2s4 -1.734340 1.018595 1.145354 -0.112782 0.590931 1.525264 -1.583458 0.948296 1.711134 0.053377 ... 0.133120 0.159131 1.151579 -0.270983 -0.793256 -0.395818 -0.000882 -0.871829 -0.321323 0.475428
r3s0 -0.783272 0.549246 0.912964 -0.846065 -0.347537 0.244262 -1.921135 0.290666 1.023833 0.347094 ... -0.121621 -0.236661 0.837270 -1.076396 -0.592567 -2.097988 -0.013013 0.231985 -1.345679 2.884452
r3s1 0.290483 0.363594 2.177999 -0.156987 -0.939555 0.443229 1.458815 2.574376 -1.076518 -0.363016 ... 1.702962 1.722178 0.999881 0.184068 -0.614013 -0.598080 -0.728771 -0.300455 1.025306 -0.240906
r3s2 -1.332260 -0.925884 -0.835553 -0.015138 1.851840 0.090223 0.281905 -0.100410 0.095656 -1.359204 ... 0.326450 -0.434180 0.019184 1.248174 -0.215904 0.850511 0.770674 0.306296 1.293982 -1.435372
r3s3 0.200371 -0.846563 1.856579 -0.600437 0.725963 0.383318 0.835080 -0.656290 -0.978341 0.180849 ... -0.889134 -0.567104 0.826408 -0.337061 -0.099261 -0.017822 -0.381713 0.862280 -0.116477 -1.101553
r3s4 -0.753709 -0.070955 -0.344824 -0.385953 -0.344296 -2.013347 0.147098 -0.802810 -1.089017 -0.130341 ... 0.140231 -0.945284 1.047795 1.883125 -0.141678 -0.783196 1.150965 -0.080518 -1.244698 1.317814

20 rows × 400 columns

Now we can start to use these data structures in the CCDimage code. It is important to note, that the values can be complex too!


In [17]:
# complex example:
df3 = pd.DataFrame(np.random.randn(20,400) + 0.1*np.random.randn(20,400)*1j, index=rows, columns=1.23*np.arange(400))

In [18]:
df3


Out[18]:
0.0 1.23 2.46 3.69 4.92 6.15 7.38 8.61 9.84 11.07 ... 479.7 480.93 482.16 483.39 484.62 485.85 487.08 488.31 489.54 490.77
r0s0 (-1.23576271913-0.185847883574j) (0.0529216538903+0.129158233718j) (0.724415189064-0.104322240946j) (-1.65301916201+0.0720106546298j) (-0.141481016733+0.0361646764067j) (-0.222757669741+0.0400796456564j) (-0.21361548993-0.0837229942064j) (0.943233544682-0.140260846666j) (0.567894567351+0.0306696240982j) (-0.902016328789+0.114131753957j) ... (-1.85902244304+0.0287101151056j) (0.921572078908-0.0710246678393j) (1.10489339086-0.0788672179216j) (-0.612657340247+0.0228959362189j) (-1.1688038002-0.156471551294j) (-0.512148594723-0.0535613010007j) (-0.756125963407-0.044222458599j) (-0.0407041390489+0.0460788111033j) (0.534728902924-0.0871808137524j) (0.132200155724+0.112614834113j)
r0s1 (-0.981939512432-0.0248213642288j) (0.201676834287+0.0332533538591j) (0.0949735642599-0.0258463075781j) (0.689998693135-0.0221663676157j) (0.830687375073-0.122014801224j) (-0.0578329368654+0.000577935992641j) (-0.422836392466-0.00451085096533j) (1.89934636394+0.145107214683j) (-1.30492029544-0.0293904757063j) (0.41542992756+0.0502382949271j) ... (-0.907980881357-0.00268087934794j) (-1.52999786965+0.100707843497j) (-0.209042624375-0.0286618206328j) (1.7737107014-0.0612318574594j) (1.56208020143-0.047036681766j) (-0.113889705226-0.0855435839874j) (2.01444713869+0.138536183898j) (-0.00361846728525+0.0192697989978j) (-2.2019193176+0.0904283692218j) (1.89656761249+0.0496429097914j)
r0s2 (1.82821071571-0.128909197042j) (0.282695903252+0.242320343889j) (0.872022068555-0.038784238982j) (-0.260669350793+0.0452001076767j) (-1.03357135086+0.0921465638993j) (0.993268356513-0.0285577214136j) (0.664639869899+0.0123485504631j) (-0.308572119062+0.0510162819116j) (-1.3338067307+0.0699766785877j) (0.43264868511+0.249764435176j) ... (1.35273043652+0.16129558619j) (0.682096489939+0.0635030727058j) (0.939696576024+0.0486226908048j) (0.116823836128-0.0815454113263j) (0.159117521664-0.0160049236552j) (0.880833567556-0.0983183596752j) (-1.34766497156-0.0644888627501j) (0.203560604553-0.0904372028935j) (-0.902965768874+0.101847259066j) (-1.68656584589-0.34389148341j)
r0s3 (-1.65411525983+0.0603759040104j) (0.730948292173+0.0543433215511j) (0.800166204462-0.157672412276j) (0.774865308574+0.077598975352j) (0.412742895736-0.00391215940931j) (-1.55615539458-0.0218819585597j) (1.13983471196-0.0848267311961j) (0.98468757961-0.0669251013132j) (1.18987141395-0.022651511229j) (-0.142564399548-0.00580960750606j) ... (-0.521348681699+0.0828146100835j) (0.858624116016-0.234377466052j) (1.31582349083-0.0740040124741j) (0.661047036604-0.105962870648j) (-0.296258381424-0.106262729265j) (0.585448302958+0.00873285003745j) (0.0773041569774-0.0801649117916j) (-0.298031886124-0.179873551867j) (-0.424452912506+0.00641296571024j) (1.14277935313-0.0433534432288j)
r0s4 (0.532960469669+0.060992160528j) (-0.466207894849-0.141470109493j) (-0.535656789752-0.0190094466983j) (-1.50120745986-0.0304633480535j) (-1.09560350726+0.142564263722j) (-0.827240055295-0.119204632388j) (0.470986902616-0.00493941945321j) (-0.559715502024-0.0607958298194j) (0.674368139181-0.0539698630908j) (1.81803386204-0.103103778697j) ... (-1.00228309014-0.020852228131j) (-0.73659892209-0.00914835489865j) (2.46517819721-0.0175754150102j) (0.35829962055-0.053494484444j) (-1.68000068509+0.0381826143409j) (-1.18632981202+0.0312460319758j) (-1.57648660658+0.116982019007j) (0.0592202718681+0.10224268974j) (-0.705469232509+0.188978635528j) (0.940453997001+0.200955905689j)
r1s0 (-0.0831914112863+0.163198685809j) (-0.932399221793-0.0270127921806j) (0.427573887868-0.00445047155494j) (-0.000164779002153-0.0770934811935j) (-0.60524235923+0.00705692043108j) (0.502600036119+0.0137702987694j) (0.409688265918-0.0165393442579j) (1.22242718331+0.0143142027866j) (1.94188539922-0.00801615876049j) (0.617609373478+0.0584193356061j) ... (0.490887761887+0.0561030719528j) (-1.42813701998+0.133173571147j) (0.57084612047+0.105991452859j) (1.08312328436-0.0472424968793j) (-0.774689873352-0.0762190053713j) (-0.560759666671+0.0903862374683j) (0.118198608503+0.0267582492836j) (0.788042868258-0.0725105868995j) (-0.227261419287+0.00541815982323j) (-1.25674127573-0.0267763055207j)
r1s1 (-0.319712138997-0.152039426621j) (0.962655283964-0.243310222492j) (0.320005444684-0.0870593939857j) (1.27041390617-0.0120088668168j) (-2.51556821593+0.0685790831661j) (-0.201284566126+0.0260898243789j) (2.43213174024-0.087934813983j) (-0.0996718667154+0.0786513923389j) (-0.456466319497-0.142287806773j) (-0.23715405766+0.0540842593935j) ... (1.09841370877-0.1940510802j) (-0.33820358124-0.0492889902562j) (-0.051926884372+0.0973298562776j) (1.77457072607-0.0959492043426j) (-2.38103188069+0.069932995824j) (-0.970222044917+0.00067614039896j) (-1.12675579635+0.0335868951329j) (1.53142098606+0.152792493145j) (0.213834644546-0.16115113382j) (-0.0968034623865-0.171325028243j)
r1s2 (0.831727824633-0.00318901160347j) (0.670548044311-0.125513485758j) (-2.06714582736-0.00875867430169j) (-0.654725965821+0.0119019903557j) (-0.653645883852+0.0241122954628j) (1.20828629401+0.282712860759j) (-0.63336936082-0.137097406702j) (-0.427047335452+0.0667830651225j) (-0.0543230476242+0.0381590010805j) (-0.103057538537-0.0317915405261j) ... (-1.76133869177+0.022786358294j) (-1.26405242912-0.00412356178994j) (1.20109198572-0.0125398053077j) (-2.42626474326-0.103746205012j) (0.260087881332+0.190255710124j) (0.840563908149+0.0665444871917j) (0.853656651832-0.108262152423j) (-0.097265347589-0.0666623011706j) (-0.764838279961+0.132936391357j) (-0.655976301017+0.0658942429166j)
r1s3 (-1.37525605478+0.0678864165322j) (1.39843366771+0.149282493709j) (-1.07043186243-0.138601350921j) (0.226995259392-0.170678108606j) (-0.592956598071-0.011135640728j) (-1.13633140889-0.0977114880465j) (0.71793208664-0.0472079625352j) (0.484789162651+0.0190007791447j) (0.670024169896-0.182424052248j) (-1.64848135044-0.0356758571558j) ... (-0.523358560066+0.17072233912j) (-0.0573552953067+0.0234892239801j) (0.562182689716+0.154433036795j) (-1.09795087178+0.160299537214j) (-1.26272257316+0.0405699118969j) (1.0112775784-0.151859713973j) (1.34656374214+0.0209350360603j) (-0.352742637211+0.0387787941876j) (-2.10893766757+0.0289713708344j) (0.564553178309+0.0892860029922j)
r1s4 (-1.35864957725-0.0447250801488j) (0.779966888467+0.0136621556124j) (0.928722579336+0.0239251376435j) (0.704362820544-0.100896397765j) (1.11845252976-0.0201786108752j) (1.41187663+0.0561016914286j) (0.256765284857-0.0425308436599j) (-1.12964774924-0.0493718921286j) (-0.289643117248-0.0901042557157j) (1.82986058501-0.053617359593j) ... (0.775279336714-0.0020430818306j) (0.79349394592+0.0458080908247j) (-0.636097338349-0.0223110597175j) (-0.0942871617806-0.0385916912193j) (-0.54854356993-0.123682849729j) (-0.287895794729-0.139519267459j) (-1.65203801979+0.0580922015492j) (-0.850825581176+0.0911781358634j) (1.77223438561+0.112060054574j) (-0.142146663816-0.0392379373917j)
r2s0 (0.277619243623-0.0427957406739j) (0.454509711444+0.0363555439813j) (-1.40515793167-0.0131875788012j) (-0.815207408487-0.013487555092j) (-0.879677699096-0.0322959498321j) (-0.0412191284588+0.166378236118j) (2.17966690677+0.155304972191j) (0.190455011781+0.040304323902j) (0.609729657567-0.155021391074j) (1.70810811617-0.0639315943095j) ... (1.02422026105-0.0838735917413j) (0.739515224746-0.262301328956j) (-0.00569714348009-0.0630193480716j) (0.478751031276-0.0463151243236j) (1.47355875614+0.0810156047213j) (-2.1078851683-0.126895542499j) (-0.900674407479-6.03316465891e-05j) (-0.87699862263-0.0971644616491j) (1.03894530595-0.0753399206228j) (-1.02333376986-0.0265737025497j)
r2s1 (0.524313763169+0.0189397616475j) (-0.388311328443+0.071677014751j) (-0.644299314804+0.19351122174j) (-0.541380022681-0.178178835193j) (1.02854161477+0.104390852437j) (-0.707628295546+0.116185494295j) (0.685896371347+0.242698503057j) (0.119642948333+0.178675744246j) (-2.010728999-0.0271840451421j) (1.02711500043+0.0220540172164j) ... (-0.606532092063+0.177257146212j) (0.979674588224+0.157539648066j) (0.929940730983-0.0275762229916j) (0.141661562777-0.176059043843j) (-0.0226442496019-0.00766434004073j) (0.804932713132-0.0197916726564j) (-0.774998828128-0.109170097404j) (-1.21500939419-0.0226419409204j) (0.432351070057-0.112096207893j) (-0.623104127852+0.162751485932j)
r2s2 (0.19191303557-0.00443282551489j) (1.09113477106-0.0290818741361j) (-0.318858525415+0.0539055964992j) (1.46363984723-0.178994583996j) (-0.477411909318-0.0973601218972j) (0.961024377412+0.0610789859716j) (-1.6549532995+0.110539365688j) (0.265897462138+0.0369672524016j) (1.67912122761+0.122387494111j) (-0.934075683675-0.00994285460239j) ... (0.952272211831+0.0191264579956j) (1.48656439513+0.020229068593j) (1.08380360247+0.0721547133717j) (-0.545582249859-0.0144008488916j) (0.35824142991-0.181217990671j) (0.608126477974+0.0126074757375j) (0.504841780735+0.0357286720151j) (-0.318060048709-0.129518631333j) (-1.92077387202-0.00295015613251j) (1.23116071305-0.077373931922j)
r2s3 (0.193914147223-0.0685759408907j) (-0.0107207597223+0.0360968517764j) (0.0518486323549+0.0958714907186j) (-0.697514531795-0.222074097922j) (0.842644278679-0.0817592408113j) (-0.314372692329-0.0397710203659j) (1.01161575823+0.0342086303863j) (1.94968533822-0.0880207989501j) (-1.16107622524+0.0445044977446j) (-0.578112928577-0.255515206264j) ... (0.950842321374+0.0256744399117j) (-1.59482251819-0.0310176505923j) (2.40615779281+0.016482678391j) (1.44914992112-0.0825611459205j) (-1.67706630796-0.0991907682338j) (-0.406479825809-0.0474725552964j) (-0.103273724421+0.0318583452567j) (-0.82176688448+0.105537370183j) (1.04343824798+0.216775914389j) (-0.269981013517+0.0757249971957j)
r2s4 (1.26312298098+0.0353078386081j) (0.0533283352593+0.10201972423j) (1.60921928791-0.236581622652j) (0.522521249696+0.0108511353459j) (-0.967412770576+0.0812200386155j) (-0.193614131481-0.0774930960243j) (0.9006873442+0.131892223023j) (-0.357434703123+0.0680903962007j) (0.508501337968-0.166044855544j) (-0.962241021062-0.0554235959826j) ... (0.309937655799+0.229761610388j) (0.578970754697+0.138883337038j) (0.0562531454811+0.0570736542305j) (0.678386157803-0.0169637315075j) (0.299387418537-0.0813273573021j) (-0.924431274846+0.102748278726j) (-0.895829098463+0.0463847585675j) (0.393481278578-0.0160656428025j) (0.346289438182-0.0346145319213j) (-0.242447949706-0.0450311580246j)
r3s0 (0.373796254925+0.154708294713j) (-1.31989898631+0.0443768735327j) (0.90529876174+0.0212612189138j) (0.61297741336-0.15236636986j) (-0.395771039549-0.14107214228j) (-0.818347497072+0.114746699381j) (-0.843940397708+0.0106815881074j) (-0.00297870741756-0.0341999546688j) (-1.06812694201+0.0144676965427j) (-1.25812808785-0.0954030642623j) ... (-0.0469815160827+0.0232703058459j) (0.24470416322+0.164586916579j) (1.95366721279-0.169896505798j) (1.16027384463-0.0734668213404j) (-0.202029190031-0.160334075452j) (1.24668565404+0.126363573094j) (0.999094720001+0.0595224992118j) (1.33229853348-0.00655201071896j) (1.62795450112+0.0756362616671j) (-2.53887641653+0.0571107676056j)
r3s1 (-1.97452985187-0.00917296924165j) (-0.044038492442-0.0533890772792j) (0.0986571181164+0.0710607280918j) (-1.1207128996+0.0586751903985j) (-1.5802345318-0.0334258835107j) (0.12135481213-0.0837061277687j) (-0.822756383009+0.0918955806026j) (0.262795552307-0.163477735507j) (0.468747437275+0.107957622074j) (-1.28757025168+0.062082210564j) ... (0.324517453127-0.0994012120212j) (-0.726165686889-0.0214926813727j) (0.670027504726-0.067414825192j) (0.672493610096+0.0259248657871j) (1.15051072605-0.0835546880332j) (-1.38438661204+0.0219637888629j) (-0.00436738953142+0.0626452726052j) (-0.250798820508+0.0744165104815j) (1.2172410505-0.0602919040257j) (1.20402719323+0.0942937778706j)
r3s2 (-0.733355394908+0.056188510842j) (-0.296812258943-0.026928650419j) (2.09901585316+0.0526436018032j) (0.182775108299+0.115145887806j) (-1.1379775679-0.0759761080345j) (-1.14674798938-0.0278691624333j) (0.239838433415+0.0632058032046j) (-0.677137889436-0.0456090271711j) (0.489696281226-0.0696107586012j) (-0.205295893752+0.0645651443053j) ... (0.927739415542+0.0779916955444j) (0.214939857223+0.116549139614j) (-0.379743555744+0.0572046696832j) (-0.373751223244-0.212776530072j) (0.308789167635-0.182568076047j) (0.948144120608-0.124195507151j) (-1.15062672337-0.0303554524768j) (1.71342836304-0.0034973810374j) (0.425120909614-0.0466252376535j) (0.815838070076+0.00567032218644j)
r3s3 (-0.204728653523-0.0770652212807j) (-0.239887247958-0.0825695492337j) (-0.214408098315-0.0638810016771j) (0.792915996552-0.190837703973j) (1.12443529821+0.0606369062219j) (-0.232448865638-0.0762480375094j) (0.260266836464+0.13866206314j) (1.54180710593-0.0523163676016j) (0.970536907823-0.144805603398j) (-0.237802580149-0.00572604234727j) ... (0.402640963686-0.02172174908j) (1.25094400433+0.013017533622j) (-1.50153426709-0.145616537321j) (0.418055057174+0.103958323504j) (-0.296678124153-0.109653945176j) (1.4713497838-0.0613489063866j) (-0.537638774566+0.130145597662j) (-0.475989304026-0.114262040163j) (0.726076244451-0.0957531986279j) (-0.132013018173-0.0619341538463j)
r3s4 (1.05103685387-0.189742584952j) (-0.0509470053352+0.142728853561j) (-1.79861171511+0.16745376313j) (-0.945647204637+0.1164464946j) (-0.760497351997+0.0301875360999j) (1.22773767134+0.06532550402j) (0.881494668841-0.0154188411389j) (0.223888650066+0.104219802766j) (-1.83339245782-0.231777846886j) (0.426401310564+0.0241752261768j) ... (-1.31086262875+0.0397126601754j) (0.468717155464+0.0327314502877j) (-0.748550410265-0.0308808654089j) (-1.49144037642+0.00352974857982j) (-3.18461600504-0.0270045225108j) (0.312807130525+0.191219919879j) (-1.97125711411-0.0765881341189j) (-1.49054702473-0.202426634312j) (-0.827802382667-0.00395477668367j) (0.955755828135-0.0491122893626j)

20 rows × 400 columns

slicing a dataframe:

We make frequent use of array slicing in python to access portions of our data array or to stack and analyze different parts of the data. Now we'll tinker on the dataframe to find out the equivalent methods.

First, the [] operation slices by rows (can use their names):


In [19]:
df3["r3s1":"r3s4"]


Out[19]:
0.0 1.23 2.46 3.69 4.92 6.15 7.38 8.61 9.84 11.07 ... 479.7 480.93 482.16 483.39 484.62 485.85 487.08 488.31 489.54 490.77
r3s1 (-1.97452985187-0.00917296924165j) (-0.044038492442-0.0533890772792j) (0.0986571181164+0.0710607280918j) (-1.1207128996+0.0586751903985j) (-1.5802345318-0.0334258835107j) (0.12135481213-0.0837061277687j) (-0.822756383009+0.0918955806026j) (0.262795552307-0.163477735507j) (0.468747437275+0.107957622074j) (-1.28757025168+0.062082210564j) ... (0.324517453127-0.0994012120212j) (-0.726165686889-0.0214926813727j) (0.670027504726-0.067414825192j) (0.672493610096+0.0259248657871j) (1.15051072605-0.0835546880332j) (-1.38438661204+0.0219637888629j) (-0.00436738953142+0.0626452726052j) (-0.250798820508+0.0744165104815j) (1.2172410505-0.0602919040257j) (1.20402719323+0.0942937778706j)
r3s2 (-0.733355394908+0.056188510842j) (-0.296812258943-0.026928650419j) (2.09901585316+0.0526436018032j) (0.182775108299+0.115145887806j) (-1.1379775679-0.0759761080345j) (-1.14674798938-0.0278691624333j) (0.239838433415+0.0632058032046j) (-0.677137889436-0.0456090271711j) (0.489696281226-0.0696107586012j) (-0.205295893752+0.0645651443053j) ... (0.927739415542+0.0779916955444j) (0.214939857223+0.116549139614j) (-0.379743555744+0.0572046696832j) (-0.373751223244-0.212776530072j) (0.308789167635-0.182568076047j) (0.948144120608-0.124195507151j) (-1.15062672337-0.0303554524768j) (1.71342836304-0.0034973810374j) (0.425120909614-0.0466252376535j) (0.815838070076+0.00567032218644j)
r3s3 (-0.204728653523-0.0770652212807j) (-0.239887247958-0.0825695492337j) (-0.214408098315-0.0638810016771j) (0.792915996552-0.190837703973j) (1.12443529821+0.0606369062219j) (-0.232448865638-0.0762480375094j) (0.260266836464+0.13866206314j) (1.54180710593-0.0523163676016j) (0.970536907823-0.144805603398j) (-0.237802580149-0.00572604234727j) ... (0.402640963686-0.02172174908j) (1.25094400433+0.013017533622j) (-1.50153426709-0.145616537321j) (0.418055057174+0.103958323504j) (-0.296678124153-0.109653945176j) (1.4713497838-0.0613489063866j) (-0.537638774566+0.130145597662j) (-0.475989304026-0.114262040163j) (0.726076244451-0.0957531986279j) (-0.132013018173-0.0619341538463j)
r3s4 (1.05103685387-0.189742584952j) (-0.0509470053352+0.142728853561j) (-1.79861171511+0.16745376313j) (-0.945647204637+0.1164464946j) (-0.760497351997+0.0301875360999j) (1.22773767134+0.06532550402j) (0.881494668841-0.0154188411389j) (0.223888650066+0.104219802766j) (-1.83339245782-0.231777846886j) (0.426401310564+0.0241752261768j) ... (-1.31086262875+0.0397126601754j) (0.468717155464+0.0327314502877j) (-0.748550410265-0.0308808654089j) (-1.49144037642+0.00352974857982j) (-3.18461600504-0.0270045225108j) (0.312807130525+0.191219919879j) (-1.97125711411-0.0765881341189j) (-1.49054702473-0.202426634312j) (-0.827802382667-0.00395477668367j) (0.955755828135-0.0491122893626j)

4 rows × 400 columns

Select a few columns:


In [76]:
df4 = df3[df3.columns[4:10]]

Plotting

It's easy to plot using the built-in method .plot() from pandas. There are also ways to access the data and plot using matplotlib


In [75]:
onerow = df3.loc["r3s2"].apply(np.real) # take the real part to avoid issues plotting
onerow.plot()


Out[75]:
<matplotlib.axes._subplots.AxesSubplot at 0x117955cf8>

In [77]:
# can also use the regular old plot calls from matplotlib
plt.plot(np.imag(df3.loc["r3s2"]))
plt.plot(np.real(df3.loc["r3s2"]))


Out[77]:
[<matplotlib.lines.Line2D at 0x117e74208>]

In [41]:
plt.hexbin(np.real(df3.loc["r3s2"]),np.imag(df3.loc["r3s2"]))


Out[41]:
<matplotlib.collections.PolyCollection at 0x10b911160>

More stats:

We can do a number of cool things with the data frame. For example, plotting different modes against each other using the scatter_matrix function.


In [58]:
from pandas.tools.plotting import scatter_matrix

Apply

How to change values in the columns:


In [69]:
# take the real part or abs of the columns (save as a different dataframes)
df4real = df4.apply(np.real)
df4abs = df4.apply(np.abs)

In [72]:
# the diagonal will be the kernel density estimate (kde)
scatter_matrix(df4abs, figsize=(8, 8), diagonal='kde')


Out[72]:
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x1161151d0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11614ab38>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1161957b8>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1161c8c18>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1163a3b70>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1163e5c18>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x116434748>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11644aa90>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1164b5f98>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11662de10>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116672f98>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1166c2438>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x1166fa908>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11674c1d0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1167103c8>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1167d10b8>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x11691a320>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1169595f8>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x1169ab048>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1169e19b0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116a28908>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116a68a20>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116ab7470>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116aca940>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x116c3cdd8>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116c8a908>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116cca710>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116d15c50>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116d52470>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116d9f198>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x116d68828>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116e21ef0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116e6df60>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116eb3080>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x116efba90>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1161af6a0>]], dtype=object)

In [1]:
# Format the notebook using style by Lorena Barba (http://lorenabarba.com/)
# run this cell first to use the nice format.
from IPython.core.display import HTML
def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


Out[1]:

In [ ]: