Notebook for exploring the cause of airflow conn_id being truncated.

Truncating seems like a good idea for some of the fields, but for connection id it seems like the full name would be more useful.

airflow@313edb656578:~$ airflow connections --list
[2019-01-08 19:56:15,947] {{settings.py:174}} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-01-08 19:56:16,350] {{__init__.py:51}} INFO - Using executor SequentialExecutor
╒════════════════════════════════╤═════════════════════════╤════════════════════════════════╤════════╤════════════════╤══════════════════════╤════════════════════════════════╕
│ Conn Id                        │ Conn Type               │ Host                           │ Port   │ Is Encrypted   │ Is Extra Encrypted   │ Extra                          │
╞════════════════════════════════╪═════════════════════════╪════════════════════════════════╪════════╪════════════════╪══════════════════════╪════════════════════════════════╡
│ 'long_names...are_truncated'                  │ 'mysql'                 │ 'localhost'                    │ None   │ True           │ False                │ None                           │
╘════════════════════════════════╧═════════════════════════╧════════════════════════════════╧════════╧════════════════╧══════════════════════╧════════════════════════════════╛

At first I thought it was tabulate that was doing the truncating....


In [3]:
# will probably need to pip install this external to jupyter
from tabulate import tabulate
import reprlib
mydict = dict()

In [ ]:


In [4]:
def populate_array(mydict):
    for i in range(50):
        line = ''
        for j in range(50):
            line += 'X'
            mydict[j] = line

In [5]:
newdict = dict()
populate_array(newdict)

In [6]:
newdict


Out[6]:
{0: 'X',
 1: 'XX',
 2: 'XXX',
 3: 'XXXX',
 4: 'XXXXX',
 5: 'XXXXXX',
 6: 'XXXXXXX',
 7: 'XXXXXXXX',
 8: 'XXXXXXXXX',
 9: 'XXXXXXXXXX',
 10: 'XXXXXXXXXXX',
 11: 'XXXXXXXXXXXX',
 12: 'XXXXXXXXXXXXX',
 13: 'XXXXXXXXXXXXXX',
 14: 'XXXXXXXXXXXXXXX',
 15: 'XXXXXXXXXXXXXXXX',
 16: 'XXXXXXXXXXXXXXXXX',
 17: 'XXXXXXXXXXXXXXXXXX',
 18: 'XXXXXXXXXXXXXXXXXXX',
 19: 'XXXXXXXXXXXXXXXXXXXX',
 20: 'XXXXXXXXXXXXXXXXXXXXX',
 21: 'XXXXXXXXXXXXXXXXXXXXXX',
 22: 'XXXXXXXXXXXXXXXXXXXXXXX',
 23: 'XXXXXXXXXXXXXXXXXXXXXXXX',
 24: 'XXXXXXXXXXXXXXXXXXXXXXXXX',
 25: 'XXXXXXXXXXXXXXXXXXXXXXXXXX',
 26: 'XXXXXXXXXXXXXXXXXXXXXXXXXXX',
 27: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 28: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 29: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 30: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 31: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 32: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 33: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 34: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 35: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 36: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 37: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 38: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 39: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 40: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 41: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 42: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 43: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 44: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 45: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 46: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 47: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 48: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
 49: 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}

In [7]:
mylist = []
for each in newdict:
    mylist.append([newdict[each]])

In [8]:
mylist


Out[8]:
[['X'],
 ['XX'],
 ['XXX'],
 ['XXXX'],
 ['XXXXX'],
 ['XXXXXX'],
 ['XXXXXXX'],
 ['XXXXXXXX'],
 ['XXXXXXXXX'],
 ['XXXXXXXXXX'],
 ['XXXXXXXXXXX'],
 ['XXXXXXXXXXXX'],
 ['XXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'],
 ['XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX']]

In [10]:
# The trucation in airflow connections list is actually caused by use of reprlib, not tabulate
conns = [map(reprlib.repr, conn) for conn in mylist]

In [11]:
conns


Out[11]:
[<map at 0x7fde00dfaf28>,
 <map at 0x7fde00dfae80>,
 <map at 0x7fde00dfadd8>,
 <map at 0x7fde00dfac18>,
 <map at 0x7fde00dfa780>,
 <map at 0x7fde00dfaa90>,
 <map at 0x7fde00dfa9e8>,
 <map at 0x7fde00dfaac8>,
 <map at 0x7fde00dfafd0>,
 <map at 0x7fde00dfa6d8>,
 <map at 0x7fde00dfada0>,
 <map at 0x7fde00df8080>,
 <map at 0x7fde00df8128>,
 <map at 0x7fde00df81d0>,
 <map at 0x7fde00df8278>,
 <map at 0x7fde00df8320>,
 <map at 0x7fde00df83c8>,
 <map at 0x7fde00df8470>,
 <map at 0x7fde00df8518>,
 <map at 0x7fde00df85c0>,
 <map at 0x7fde00df8668>,
 <map at 0x7fde00df8710>,
 <map at 0x7fde00df87b8>,
 <map at 0x7fde00df8860>,
 <map at 0x7fde00df8908>,
 <map at 0x7fde00df89b0>,
 <map at 0x7fde00df8a58>,
 <map at 0x7fde00df8b00>,
 <map at 0x7fde00df8ba8>,
 <map at 0x7fde00df8c50>,
 <map at 0x7fde00df8cf8>,
 <map at 0x7fde00df8da0>,
 <map at 0x7fde00df8e48>,
 <map at 0x7fde00df8ef0>,
 <map at 0x7fde00df8f98>,
 <map at 0x7fde00df6080>,
 <map at 0x7fde00df6128>,
 <map at 0x7fde00df61d0>,
 <map at 0x7fde00df6278>,
 <map at 0x7fde00df6320>,
 <map at 0x7fde00df63c8>,
 <map at 0x7fde00df6470>,
 <map at 0x7fde00df6518>,
 <map at 0x7fde00df65c0>,
 <map at 0x7fde00df6668>,
 <map at 0x7fde00df6710>,
 <map at 0x7fde00df67b8>,
 <map at 0x7fde00df6860>,
 <map at 0x7fde00df6908>,
 <map at 0x7fde00df69b0>]

In [ ]:


In [12]:
print (tabulate(conns, tablefmt="fancy_grid"))


╒════════════════════════════════╕
│ 'X'                            │
├────────────────────────────────┤
│ 'XX'                           │
├────────────────────────────────┤
│ 'XXX'                          │
├────────────────────────────────┤
│ 'XXXX'                         │
├────────────────────────────────┤
│ 'XXXXX'                        │
├────────────────────────────────┤
│ 'XXXXXX'                       │
├────────────────────────────────┤
│ 'XXXXXXX'                      │
├────────────────────────────────┤
│ 'XXXXXXXX'                     │
├────────────────────────────────┤
│ 'XXXXXXXXX'                    │
├────────────────────────────────┤
│ 'XXXXXXXXXX'                   │
├────────────────────────────────┤
│ 'XXXXXXXXXXX'                  │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX'                 │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXX'                │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXX'               │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXX'              │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXX'             │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXX'            │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXX'           │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXX'          │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXX'         │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXX'        │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXX'       │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXX'      │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXXX'     │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXXXX'    │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXXXXX'   │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXXXXXX'  │
├────────────────────────────────┤
│ 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
├────────────────────────────────┤
│ 'XXXXXXXXXXXX...XXXXXXXXXXXXX' │
╘════════════════════════════════╛

So it turns out that the cause of truncation is use of the reprlib library.


In [ ]: