Create a truth database

This notebook shows how to cull needed information from the opsim and cached star databases for the relevant truth information for the stars in the Twinkles field. Once the data is gathered we also calculate the expected flux for the objects in each visit in the Twinkles run. This is used to gather values to compare against the DM processed data in our pserv databases.

"True" in this case refers to the values that come from our LSST CATSIM database. This CATSIM database stores the LSST simulated universe model that we use to provide inputs to LSST simulations. It is important to note that this means that "true" does not refer to actual stars in sky, but to the known inputs to our simulations. More information on the LSST Simulations can be found here.

Note: Requires an Opsim database. Twinkles 1.1 uses kraken_1042 while Twinkles 1.3 uses minion_1016.


In [1]:
import desc.monitor
import pandas as pd
%load_ext autoreload
%autoreload 2


/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sims_catalogs/2.3.3.3.sims+5/python/lsst/sims/catalogs/db/dbConnection.py:470: UserWarning: Duplicate object type id 25 specified: 
Output object ids may not be unique.
This may not be a problem if you do not want globally unique id values
  'want globally unique id values')
/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sims_catalogs/2.3.3.3.sims+5/python/lsst/sims/catalogs/db/dbConnection.py:470: UserWarning: Duplicate object type id 40 specified: 
Output object ids may not be unique.
This may not be a problem if you do not want globally unique id values
  'want globally unique id values')
/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sims_catalogs/2.3.3.3.sims+5/python/lsst/sims/catalogs/db/dbConnection.py:470: UserWarning: Duplicate object type id 4 specified: 
Output object ids may not be unique.
This may not be a problem if you do not want globally unique id values
  'want globally unique id values')

Get cached CatSim stars

We have cached all the simulated stars in the Twinkles field in the data folder of the repository. We have written methods to interact with the database.


In [2]:
truth_db_conn = desc.monitor.StarCacheDBObj(database='../data/star_cache.db')

In [3]:
truth_db_conn.columns


Out[3]:
[('id', 'simobjid', int),
 ('raJ2000', 'ra*PI()/180.'),
 ('decJ2000', 'decl*PI()/180.'),
 ('glon', 'gal_l*PI()/180.'),
 ('glat', 'gal_b*PI()/180.'),
 ('properMotionRa', '(mura/(1000.*3600.))*PI()/180.'),
 ('properMotionDec', '(mudecl/(1000.*3600.))*PI()/180.'),
 ('parallax', 'parallax*PI()/648000000.'),
 ('galacticAv', 'CONVERT(float, ebv*3.1)'),
 ('radialVelocity', 'vrad'),
 ('variabilityParameters', 'varParamStr', str, 256),
 ('sedFilename', 'sedfilename', str, 256),
 ('simobjid', 'simobjid', int),
 ('ra', 'ra', float),
 ('decl', 'decl', float),
 ('magNorm', 'magNorm', float),
 ('mura', 'mura', float),
 ('mudecl', 'mudecl', float),
 ('ebv', 'ebv', float),
 ('vrad', 'vrad', float),
 ('varParamStr', 'varParamStr', str, 256),
 ('sedfilename', 'sedfilename', str, 256),
 ('gmag', 'gmag', float)]

Use Opsim to calculate expected flux in visits.

We also have methods to interact with the LSST Opsim databases. This can be used to get the observing conditions simulated for each visit. We then use this information to create a pandas dataframe with the expected flux based upon the Opsim information and CatSim flux of the objects in the field.


In [4]:
worker = desc.monitor.TrueStars(truth_db_conn, '../../kraken_1042_sqlite.db')

In [5]:
# Just use one visit here for the demo. Otherwise it will use the list in data/selectedVisits.csv
star_df = worker.get_true_stars(for_obsHistIds=200, catalog_constraints='gmag > 11')


Generated 1 out of 1 obs_metadata
Generated fluxes for 1 out of 1 visits
/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sqlalchemy/1.0.8.lsst3+3/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py:3779: SAWarning: Textual SQL expression 'ra between 52.671111 and ...' should be explicitly declared as text('ra between 52.671111 and ...') (this warning may be suppressed after 10 occurrences)
  {"expr": util.ellipses_string(element)})
/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sqlalchemy/1.0.8.lsst3+3/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py:3779: SAWarning: Textual SQL expression 'gmag > 11' should be explicitly declared as text('gmag > 11') (this warning may be suppressed after 10 occurrences)
  {"expr": util.ellipses_string(element)})
/global/common/cori/contrib/lsst/lsstDM/twinkles_20170208/Linux64/sqlalchemy/1.0.8.lsst3+3/lib/python/SQLAlchemy-1.0.8-py2.7-linux-x86_64.egg/sqlalchemy/sql/sqltypes.py:565: SAWarning: Dialect sqlite+pysqlite does *not* support Decimal objects natively, and SQLAlchemy must convert from floating point - rounding errors and other issues may occur. Please consider storing Decimal numbers as strings or integers on this platform for lossless storage.
  'storage.' % (dialect.name, dialect.driver))

In [6]:
star_df


Out[6]:
uniqueId ra dec filter true_flux true_flux_error obsHistId
0 16285577220 52.903412 -27.189153 r 0.117658 0.040271 200
1 16288599044 52.964837 -27.207770 r 0.234963 0.040543 200
2 16291377156 52.779785 -27.259185 r 0.901140 0.042051 200
3 16295112708 53.016603 -27.257971 r 0.628290 0.041440 200
4 16311696388 52.957553 -27.413504 r 0.031243 0.040070 200
5 16314314756 52.691828 -27.473861 r 1.783972 0.043971 200
6 16332994564 53.160574 -27.570433 r 0.571313 0.041311 200
7 16342631428 53.017424 -27.670135 r 0.387217 0.040892 200
8 16346230788 52.930067 -27.712621 r 0.028436 0.040063 200
9 48113897476 53.128940 -27.288412 r 0.338467 0.040781 200
10 48114528260 53.052977 -27.316177 r 0.024397 0.040054 200
11 48114990084 53.016722 -27.334360 r 1.723779 0.043843 200
12 48117544964 53.302465 -27.359595 r 0.109794 0.040253 200
13 48121656324 52.861724 -27.527492 r 0.189955 0.040439 200
14 66464945156 52.971623 -27.222488 r 0.136568 0.040315 200
15 66468398084 53.097341 -27.257617 r 0.062685 0.040143 200
16 66468912132 52.755636 -27.313961 r 0.163597 0.040378 200
17 66470010884 53.000534 -27.295357 r 0.068206 0.040156 200
18 66471763972 52.716460 -27.362527 r 0.231929 0.040536 200
19 66474412036 52.982696 -27.363879 r 0.072997 0.040167 200
20 66474603524 53.251481 -27.328256 r 0.026841 0.040060 200
21 66477314052 53.069683 -27.393288 r 0.113042 0.040260 200
22 66477441028 52.952699 -27.411811 r 0.197153 0.040455 200
23 66478898180 52.819304 -27.452993 r 0.112149 0.040258 200
24 66481695748 53.225366 -27.436162 r 0.062434 0.040143 200
25 66481954820 53.087065 -27.459274 r 0.016569 0.040036 200
26 66483836932 53.224141 -27.468498 r 0.026376 0.040058 200
27 66485227524 52.948640 -27.527227 r 0.024176 0.040053 200
28 66487078916 53.133022 -27.528515 r 0.030999 0.040069 200
29 66488325124 52.732629 -27.601900 r 0.106993 0.040246 200
... ... ... ... ... ... ... ...
3117 1704265631748 53.144258 -27.660735 r 0.010689 0.040022 200
3118 1704265689092 53.196187 -27.654070 r 706.110140 0.365633 200
3119 1704265694212 53.034623 -27.676120 r 0.028021 0.040062 200
3120 1704265820164 52.995043 -27.682587 r 59.779923 0.113059 200
3121 1704266146820 53.109969 -27.669384 r 0.024333 0.040054 200
3122 1704266284036 52.912865 -27.697141 r 0.013705 0.040029 200
3123 1704266636292 53.199683 -27.661367 r 0.039154 0.040088 200
3124 1704267128836 53.060519 -27.684377 r 0.179119 0.040414 200
3125 1704267402244 53.195267 -27.668474 r 105.172347 0.145855 200
3126 1704267474948 52.932267 -27.704666 r 0.078243 0.040179 200
3127 1704267736068 53.009690 -27.696183 r 0.037144 0.040084 200
3128 1704267787268 53.096845 -27.684747 r 0.081768 0.040188 200
3129 1704268048388 53.167922 -27.677068 r 0.190811 0.040441 200
3130 1704268207108 52.909129 -27.713527 r 0.417539 0.040962 200
3131 1704268235780 53.063672 -27.692869 r 0.207066 0.040478 200
3132 1704268429316 53.199556 -27.676115 r 0.016080 0.040034 200
3133 1704268432388 53.028892 -27.699288 r 0.012170 0.040025 200
3134 1704268437508 53.086405 -27.691529 r 0.019795 0.040043 200
3135 1704268612612 53.067192 -27.695515 r 0.823995 0.041879 200
3136 1704269058052 53.113112 -27.693035 r 0.097796 0.040225 200
3137 1704269612036 53.083939 -27.701360 r 0.262544 0.040606 200
3138 1704270601220 52.952027 -27.727158 r 0.091358 0.040210 200
3139 1704270780420 53.115725 -27.706401 r 0.959799 0.042182 200
3140 1704271237124 53.093628 -27.712925 r 0.025498 0.040056 200
3141 1704271460356 52.978257 -27.730338 r 0.011973 0.040025 200
3142 1704271574020 52.975875 -27.731543 r 0.030166 0.040067 200
3143 1704271690756 53.019429 -27.726538 r 11.525686 0.061284 200
3144 1704271807492 52.973716 -27.733775 r 0.234004 0.040540 200
3145 1704272300036 53.003486 -27.733958 r 0.026461 0.040059 200
3146 1704272881668 53.079486 -27.728487 r 0.012434 0.040026 200

3147 rows × 7 columns

Save to a sqlite database

Finally, we save this to a sqlite database to use in our comparisons to processed data from the simulated Twinkles images.


In [7]:
worker.write_to_db(star_df, 'twinkles_truth_demo.db', if_exists='replace')

In [ ]: