notebook.community

Edit and run

This notebook will show how to use the gel package to perform group elastic net regression. Suppose we have feature matrices $A_j$ ($j = 1 \dots p$), each of size $m \times n_j$ (i.e. $m$ $n_j$-dimensional samples), and targets $y$ (a $m$ dimensional vector). Group elastic net finds coefficients $\beta_j$ ($n_j$ dimensional vector), and $\beta_0$ (scalar) by solving the optimization problem

$$ \min_{\beta_0,\dots,\beta_p} \frac{1}{2}\left\|y - \beta_0 - \sum_{j=1}^p A_j\beta_j\right\|^2 + m\sum_{j=1}^p \sqrt{n_j}\left(\lambda_1\|\beta_j\| + \lambda_2\|\beta_j\|^2\right). $$

We will test this with some generated data.





In [ ]:



    


m_tr = 100  # number of samples
ns = [10, 20, 5, 200, 1, 20]  # group sizes

# data distribution parameters
X_scale = 2.0
X_bias = 3.0

# coefficient distribution parameters
β_scale = 0.125
β_bias = 6.0

# true support
support = [0, 1, 0, 1, 1, 0]



    











In [ ]:



    


# Generate data
import torch

A_trs = [torch.randn(m_tr, n_j)*X_scale + X_bias for n_j in ns]
βs = [s_j*(torch.randn(n_j, 1)*β_scale + β_bias) for s_j, n_j in zip(support, ns)]
y_tr = sum([A_j@β_j for A_j, β_j in zip(A_trs, βs)])[:, 0]

We will now use gel to solve group elastic net with a single pair of regularization values. There are two steps to this. First, make_A is used to convert the features as needed by the library. This, and other parameters are passed to gel_solve.

There are two implementations, gelfista and gelcd, which use proximal gradient descent and coordinate descent respectively. Refer to the respective source files for details on these methods. Here, we will use gelfista. gelcd is used in the same way.





In [ ]:



    


from gel.gelfista import make_A, gel_solve



    











In [ ]:



    


A_conv_tr = make_A(
    A_trs,
    torch.tensor(ns),  # note that this needs to be a LongTensor
    device=torch.device("cpu"),
    dtype=torch.float32,
)



    











In [ ]:



    


print(A_conv_tr.shape)



    


















    






torch.Size([6, 200, 100])

















In [ ]:



    


print(y_tr.shape)  # note that y should be one dimensional



    


















    






torch.Size([100])

















In [ ]:



    


l_1 = 1.0  # λ_1
l_2 = 0.1  # λ_2



    











In [ ]:



    


b_0, B = gel_solve(
    A_conv_tr,
    y_tr,
    l_1,
    l_2,
    ns,
    b_init=None,     # initialization value - useful if solving multiple instances
    t_init=None,     # initial step size for line search - default (None) is to use previous iteration's value
    ls_beta=0.99,    # line search shrinkage factor - default (None) is to not perform line search and use t_init
    max_iters=5000,  # maximum iterations - default (None) is unlimited
    rel_tol=1e-5,    # relative change in b_0, B to stop the algorithm (default 1e-6)
    verbose=True,    # verbosity (default False)
)



    


















    






Solving gel with FISTA (l_1 1, l_2 0.1): 5000it [00:39, 126.80it/s, t=1.1e-09, rel change=5.3e-05]

The returned b_0 is a scalar, and B is a matrix. Each row of B is the coefficient vector for the corresponding group, padded with 0s. So we will first extract the vectors.





In [ ]:



    


βhats = [B[j, :ns[j]] for j in range(len(ns))]



    











In [ ]:



    


print(b_0)



    


















    






1726.2655029296875

















In [ ]:



    


print([βhat_j.norm() for βhat_j in βhats])



    


















    






[tensor(11.4616), tensor(26.9095), tensor(14.2158), tensor(44.3119), tensor(10.5515), tensor(12.6890)]

This is the solution for a single (l_1, l_2) pair. But practically, we want to use a grid of values. So now, we will use the gelpaths module to sweep over multiple regularization values and pick the best based on validation results.





In [ ]:



    


# First, we generate validation data.
m_val = 30
A_vals = [torch.randn(m_val, n_j)*X_scale + X_bias for n_j in ns]
y_val = sum([A_j@β_j for A_j, β_j in zip(A_vals, βs)])[:, 0]



    











In [ ]:



    


# We need to standardize the data to prevent repeated computations.
A_tr_μs = [A_j.mean(dim=0, keepdim=True) for A_j in A_trs]
A_tr_σs = [A_j.std(dim=0, keepdim=True) for A_j in A_trs]
A_tr_stans = [(A_j - A_μ_j) / A_σ_j for A_j, A_μ_j, A_σ_j in zip(A_trs, A_tr_μs, A_tr_σs)]
A_val_stans = [(A_j - A_μ_j) / A_σ_j for A_j, A_μ_j, A_σ_j in zip(A_vals, A_tr_μs, A_tr_σs)]

y_tr_μ = y_tr.mean()
y_tr_σ = y_tr.std()
y_tr_stan = (y_tr - y_tr_μ) / y_tr_σ

gelpaths follows a 2-step procedure. For each (l_1, l_2) pair, the corresponding group elastic net problem is solved; this is then used to find the support by finding non-zero coefficient vectors. The support is then used to solve ridge regression over a range of regularization values.

For each ridge solution, a summary function is called. This is where we will compute the validation error, to find the optimal regularization values. Note that the summary function expects the features as a single matrix.





In [ ]:



    


X_val = torch.cat(A_val_stans, dim=1).t()  # note the transpose

def summary(_support, _b):
    if _support is None:
        # empty support, so just use the computed training mean
        yhat_val = y_tr_μ
        _support_set = {}
    else:
        X_val_supp = X_val[_support]
        yhat_val = (X_val_supp.t() @ _b) * y_tr_σ + y_tr_μ  # we have to rescale
        _support_set = set([s.item() for s in _support])
        
    rmse = ((y_val - yhat_val) ** 2).mean().sqrt().item()

    # compute group support
    group_support = []
    start_idx = 0
    for n_j in ns:
        group_idxs = range(start_idx, start_idx + n_j)
        if all(i in _support_set for i in group_idxs):
            group_support.append(1)
        elif not any(i in _support_set for i in group_idxs):
            group_support.append(0)
        else:
            # This shouldn't happen! Groups are either all in or all out.
            raise AssertionError
        start_idx += n_j

    return group_support, rmse

There are two implementations in gelpaths. gel_paths requires manually specifying regularization values, and gel_paths2 picks them automatically. We will use gel_paths2 here.





In [ ]:



    


from gel.gelpaths import gel_paths2

l_rs = torch.logspace(-10, 10, 100).tolist()  # regularization strengths for ridge regression

summaries = gel_paths2(
    gel_solve,
    {
        # arguments to gel_solve
        "t_init": None,
        "ls_beta": 0.99,
        "max_iters": 3000,
        "rel_tol": 1e-5,
    },
    make_A,                                   # we don't explicitly call make_A here
    A_tr_stans,                               # we need to pass the standardized A values
    y_tr_stan,                                # note: standardized ouput vector
    ks=[0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 0.99], # this is a list of values to trade-off
                                              #   between l_1 and l_2 regularizations, with
                                              #   l_1 = k*l and l_2 = (1 - k)*l
    n_ls=50,                                  # number of l values for each k value
    l_eps=1e-5,                               # ratio of minimum to maximum l value;
                                              #    n_ls number of l values are generated
                                              #    according to this ratio, for each k
    l_rs=l_rs,                                # ridge regularization values
    summ_fun=summary,                         # the summary function
    supp_thresh=1e-4,                         # norm threshold for considering coefficient vectors
                                              #     to be 0 (default 1e-6)
    device=torch.device("cpu"),               # pytorch device (default cpu)
    verbose=True,                             # verbosity (default False)
    ls_grid=None,                             # override l value computation with a fixed grid
                                              #     (default is None)
    aux_rel_tol=1e-3,                         # tolerance for solving auxiliary problems
                                              #     to get better initializations (default 1e-3)
    dtype=torch.float32,                      # torch dtype (default float32)
)



    


















    






Solving gel with FISTA (l_1 0.21, l_2 21): 595it [00:04, 140.07it/s, t=0.079, rel change=2.1e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 5827.85it/s]

Solving gel with FISTA (l_1 0.17, l_2 16): 4it [00:00, 13.43it/s, t=0.13, rel change=3.7e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2775.79it/s]

Solving gel with FISTA (l_1 0.13, l_2 13): 4it [00:00,  5.55it/s, t=0.015, rel change=1.9e-07]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 6474.59it/s]

Solving gel with FISTA (l_1 0.1, l_2 10): 5it [00:00, 76.45it/s, t=0.64, rel change=6e-06]  
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 6179.73it/s]

Solving gel with FISTA (l_1 0.082, l_2 8.1): 13it [00:00, 23.02it/s, t=0.033, rel change=2.5e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 5444.53it/s]

Solving gel with FISTA (l_1 0.065, l_2 6.4): 7it [00:00,  9.56it/s, t=0.068, rel change=2.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2960.59it/s]

Solving gel with FISTA (l_1 0.051, l_2 5.1): 7it [00:00, 66.06it/s, t=0.38, rel change=5.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3455.77it/s]

Solving gel with FISTA (l_1 0.04, l_2 4): 7it [00:00, 54.39it/s, t=0.36, rel change=1.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2664.18it/s]

Solving gel with FISTA (l_1 0.032, l_2 3.2): 10it [00:00, 44.84it/s, t=0.055, rel change=9.4e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3640.19it/s]

Solving gel with FISTA (l_1 0.025, l_2 2.5): 8it [00:00, 62.92it/s, t=0.28, rel change=9.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3118.77it/s]

Solving gel with FISTA (l_1 0.02, l_2 2): 8it [00:00, 94.17it/s, t=0.35, rel change=9.1e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3417.42it/s]

Solving gel with FISTA (l_1 0.016, l_2 1.6): 12it [00:00, 96.53it/s, t=0.26, rel change=3.8e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3670.81it/s]

Solving gel with FISTA (l_1 0.012, l_2 1.2): 13it [00:00, 86.13it/s, t=0.3, rel change=8.6e-07] 
Support size: 6
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3091.80it/s]

Solving gel with FISTA (l_1 0.0099, l_2 0.98): 14it [00:00, 92.91it/s, t=0.3, rel change=1.9e-06]
Support size: 6
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1631.85it/s]

Solving gel with FISTA (l_1 0.0078, l_2 0.77): 16it [00:00, 84.97it/s, t=0.2, rel change=2.1e-06] 
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1367.65it/s]

Solving gel with FISTA (l_1 0.0062, l_2 0.61): 16it [00:00, 57.52it/s, t=0.19, rel change=6.3e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2102.08it/s]

Solving gel with FISTA (l_1 0.0049, l_2 0.48): 17it [00:00, 100.58it/s, t=0.3, rel change=6.8e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1831.50it/s]

Solving gel with FISTA (l_1 0.0039, l_2 0.38): 19it [00:00, 121.51it/s, t=0.26, rel change=6.1e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1574.63it/s]

Solving gel with FISTA (l_1 0.003, l_2 0.3): 23it [00:00, 97.12it/s, t=0.21, rel change=5.8e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1310.17it/s]

Solving gel with FISTA (l_1 0.0024, l_2 0.24): 26it [00:00, 77.62it/s, t=0.17, rel change=7.2e-06]
Support size: 256
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 681.60it/s]

Solving gel with FISTA (l_1 0.21, l_2 1.9): 604it [00:04, 144.42it/s, t=0.029, rel change=8.9e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3187.26it/s]

Solving gel with FISTA (l_1 0.17, l_2 1.5): 5it [00:00, 75.11it/s, t=0.86, rel change=2.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1494.19it/s]

Solving gel with FISTA (l_1 0.13, l_2 1.2): 5it [00:00, 102.88it/s, t=0.88, rel change=1.4e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1116.40it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.93): 11it [00:00, 87.79it/s, t=0.44, rel change=9.1e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3266.69it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.74): 15it [00:00, 103.46it/s, t=0.26, rel change=7.4e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2667.67it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.58): 16it [00:00, 109.59it/s, t=0.28, rel change=6.8e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3206.46it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.46): 18it [00:00, 95.57it/s, t=0.2, rel change=8.4e-06] 
Support size: 6
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3225.94it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.36): 18it [00:00, 64.37it/s, t=0.28, rel change=9.2e-06] 
Support size: 46
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 518.62it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.29): 21it [00:00,  5.58it/s, t=0.21, rel change=6.3e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 655.42it/s]

Solving gel with FISTA (l_1 0.025, l_2 0.23): 23it [00:00, 41.22it/s, t=0.23, rel change=9.4e-06]
Support size: 56
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 717.05it/s]

Solving gel with FISTA (l_1 0.02, l_2 0.18): 27it [00:00, 51.37it/s, t=0.15, rel change=8.2e-06]
Support size: 256
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 351.64it/s]

Solving gel with FISTA (l_1 0.21, l_2 0.49): 614it [00:05, 110.65it/s, t=0.029, rel change=5.4e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 9571.01it/s]

Solving gel with FISTA (l_1 0.17, l_2 0.39): 4it [00:00, 61.75it/s, t=0.55, rel change=4.3e-06] 
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 4294.27it/s]

Solving gel with FISTA (l_1 0.13, l_2 0.31): 4it [00:00, 116.41it/s, t=0.9, rel change=7.5e-06] 
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3958.38it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.24): 23it [00:00, 103.38it/s, t=0.24, rel change=3.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1824.41it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.19): 21it [00:00, 50.47it/s, t=0.16, rel change=4.9e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1713.68it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.15): 30it [00:00, 34.70it/s, t=0.17, rel change=6.5e-06]
Support size: 6
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1453.66it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.12): 25it [00:00, 64.09it/s, t=0.26, rel change=7e-06]  
Support size: 46
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1421.85it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.094): 38it [00:00, 115.86it/s, t=0.065, rel change=8e-06] 
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 836.64it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.075): 36it [00:00, 112.71it/s, t=0.082, rel change=9e-06] 
Support size: 256
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 709.58it/s]

Solving gel with FISTA (l_1 0.21, l_2 0.21): 975it [00:05, 179.39it/s, t=0.0066, rel change=8.4e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 9352.89it/s]

Solving gel with FISTA (l_1 0.17, l_2 0.17): 5it [00:00, 232.71it/s, t=0.96, rel change=5.4e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 4029.03it/s]

Solving gel with FISTA (l_1 0.13, l_2 0.13): 4it [00:00, 210.94it/s, t=1, rel change=7.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 4137.50it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.1): 20it [00:00, 86.72it/s, t=0.22, rel change=1.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3525.60it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.082): 26it [00:00, 102.04it/s, t=0.17, rel change=7.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2895.70it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.065): 33it [00:00, 113.94it/s, t=0.22, rel change=4.8e-06]
Support size: 26
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2135.15it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.051): 42it [00:00, 124.56it/s, t=0.15, rel change=7.1e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 845.99it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.04): 44it [00:00, 126.45it/s, t=0.17, rel change=6.5e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 807.91it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.032): 50it [00:00, 133.51it/s, t=0.18, rel change=8.4e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 850.18it/s]

Solving gel with FISTA (l_1 0.025, l_2 0.025): 60it [00:00, 149.35it/s, t=0.12, rel change=9.4e-06]
Support size: 256
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 640.72it/s]

Solving gel with FISTA (l_1 0.21, l_2 0.09): 328it [00:02, 148.94it/s, t=0.14, rel change=6.3e-06]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 7403.11it/s]

Solving gel with FISTA (l_1 0.17, l_2 0.071): 6it [00:00, 110.02it/s, t=0.81, rel change=5.5e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3020.05it/s]

Solving gel with FISTA (l_1 0.13, l_2 0.056): 5it [00:00, 170.71it/s, t=0.95, rel change=4.5e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3063.53it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.044): 22it [00:00, 93.62it/s, t=0.1, rel change=3.5e-06] 
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3526.97it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.035): 38it [00:00, 132.66it/s, t=0.24, rel change=9.1e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2849.49it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.028): 39it [00:00, 119.32it/s, t=0.19, rel change=7.7e-06]
Support size: 221
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 830.95it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.022): 44it [00:00, 133.13it/s, t=0.2, rel change=8.2e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 931.83it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.017): 49it [00:00, 143.51it/s, t=0.18, rel change=9.4e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 754.81it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.014): 69it [00:00, 123.35it/s, t=0.041, rel change=9.1e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 779.36it/s]

Solving gel with FISTA (l_1 0.025, l_2 0.011): 68it [00:00, 129.80it/s, t=0.067, rel change=9.2e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 793.98it/s]

Solving gel with FISTA (l_1 0.02, l_2 0.0086): 71it [00:00, 151.30it/s, t=0.11, rel change=9.5e-06]
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 804.30it/s]

Solving gel with FISTA (l_1 0.016, l_2 0.0068): 116it [00:00, 118.59it/s, t=0.00034, rel change=1e-05] 
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 880.40it/s]

Solving gel with FISTA (l_1 0.012, l_2 0.0053): 88it [00:00, 103.20it/s, t=0.0055, rel change=1e-05] 
Support size: 246
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 781.14it/s]

Solving gel with FISTA (l_1 0.0099, l_2 0.0042): 81it [00:00, 93.38it/s, t=0.0082, rel change=9.8e-06]
Support size: 256
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 784.26it/s]

Solving gel with FISTA (l_1 0.21, l_2 0.023): 862it [00:04, 190.53it/s, t=0.03, rel change=7.7e-06] 
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 7298.12it/s]

Solving gel with FISTA (l_1 0.17, l_2 0.018): 5it [00:00, 185.79it/s, t=1, rel change=1.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3994.58it/s]

Solving gel with FISTA (l_1 0.13, l_2 0.015): 5it [00:00, 160.32it/s, t=0.97, rel change=4.9e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 2426.88it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.011): 28it [00:00, 116.96it/s, t=0.27, rel change=2.2e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3543.08it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.0091): 34it [00:00, 149.19it/s, t=0.26, rel change=2.8e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3525.07it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.0072): 41it [00:00, 151.43it/s, t=0.2, rel change=5.3e-06]
Support size: 201
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 947.61it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.0057): 49it [00:00, 148.27it/s, t=0.18, rel change=6.3e-06]
Support size: 221
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 832.53it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.0045): 56it [00:00, 100.64it/s, t=0.15, rel change=7.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 444.14it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.0035): 70it [00:00, 62.49it/s, t=0.068, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 697.95it/s]

Solving gel with FISTA (l_1 0.025, l_2 0.0028): 69it [00:00, 147.51it/s, t=0.068, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 894.15it/s]

Solving gel with FISTA (l_1 0.02, l_2 0.0022): 69it [00:00, 109.31it/s, t=0.00071, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 924.07it/s]

Solving gel with FISTA (l_1 0.016, l_2 0.0018): 84it [00:00, 142.08it/s, t=0.023, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 836.44it/s]

Solving gel with FISTA (l_1 0.012, l_2 0.0014): 95it [00:00, 123.87it/s, t=0.0018, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 903.11it/s]

Solving gel with FISTA (l_1 0.0099, l_2 0.0011): 87it [00:00, 106.94it/s, t=0.0094, rel change=1e-05] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 688.52it/s]

Solving gel with FISTA (l_1 0.0078, l_2 0.00087): 101it [00:00, 89.18it/s, t=0.016, rel change=1e-05] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 831.48it/s]

Solving gel with FISTA (l_1 0.0062, l_2 0.00069): 73it [00:00, 117.35it/s, t=0.0025, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 778.90it/s]

Solving gel with FISTA (l_1 0.0049, l_2 0.00054): 65it [00:00, 88.88it/s, t=0.00099, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 954.65it/s]

Solving gel with FISTA (l_1 0.0039, l_2 0.00043): 59it [00:00, 49.74it/s, t=9.1e-05, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 675.61it/s]

Solving gel with FISTA (l_1 0.003, l_2 0.00034): 62it [00:00, 92.02it/s, t=0.00027, rel change=1e-05]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 960.42it/s]

Solving gel with FISTA (l_1 0.0024, l_2 0.00027): 63it [00:00, 125.32it/s, t=0.015, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 885.98it/s]

Solving gel with FISTA (l_1 0.0019, l_2 0.00021): 69it [00:00, 96.54it/s, t=0.031, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 631.67it/s]

Solving gel with FISTA (l_1 0.0015, l_2 0.00017): 28it [00:00, 54.67it/s, t=0.019, rel change=9.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 853.88it/s]

Solving gel with FISTA (l_1 0.0012, l_2 0.00013): 29it [00:00, 72.86it/s, t=0.008, rel change=9.4e-06] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 865.40it/s]

Solving gel with FISTA (l_1 0.00094, l_2 0.0001): 77it [00:00, 112.62it/s, t=0.029, rel change=1e-05] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 940.23it/s]

Solving gel with FISTA (l_1 0.00074, l_2 8.3e-05): 39it [00:00, 90.04it/s, t=0.018, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 907.40it/s]

Solving gel with FISTA (l_1 0.00059, l_2 6.5e-05): 31it [00:00, 101.12it/s, t=0.074, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1003.78it/s]

Solving gel with FISTA (l_1 0.00047, l_2 5.2e-05): 32it [00:00, 107.36it/s, t=0.12, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 879.96it/s]

Solving gel with FISTA (l_1 0.00037, l_2 4.1e-05): 25it [00:00, 92.32it/s, t=0.09, rel change=1e-05]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 901.20it/s]

Solving gel with FISTA (l_1 0.00029, l_2 3.2e-05): 26it [00:00, 91.20it/s, t=0.1, rel change=9.8e-06] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 867.59it/s]

Solving gel with FISTA (l_1 0.00023, l_2 2.6e-05): 27it [00:00, 106.69it/s, t=0.16, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 836.08it/s]

Solving gel with FISTA (l_1 0.00018, l_2 2e-05): 15it [00:00, 94.00it/s, t=0.16, rel change=8.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 839.97it/s]

Solving gel with FISTA (l_1 0.00014, l_2 1.6e-05): 14it [00:00, 87.89it/s, t=0.18, rel change=8.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 935.07it/s]

Solving gel with FISTA (l_1 0.00011, l_2 1.3e-05): 13it [00:00, 71.42it/s, t=0.18, rel change=8.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 571.00it/s]

Solving gel with FISTA (l_1 9e-05, l_2 1e-05): 12it [00:00, 62.37it/s, t=0.14, rel change=8.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 996.74it/s]

Solving gel with FISTA (l_1 7.1e-05, l_2 7.9e-06): 11it [00:00, 91.51it/s, t=0.21, rel change=9.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 864.25it/s]

Solving gel with FISTA (l_1 5.6e-05, l_2 6.2e-06): 10it [00:00, 64.64it/s, t=0.12, rel change=8.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1041.04it/s]

Solving gel with FISTA (l_1 4.4e-05, l_2 4.9e-06): 9it [00:00, 70.36it/s, t=0.2, rel change=8.9e-06] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 849.79it/s]

Solving gel with FISTA (l_1 3.5e-05, l_2 3.9e-06): 9it [00:00, 70.63it/s, t=0.23, rel change=8.5e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 827.12it/s]

Solving gel with FISTA (l_1 2.8e-05, l_2 3.1e-06): 7it [00:00, 61.25it/s, t=0.24, rel change=8.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 898.21it/s]

Solving gel with FISTA (l_1 2.2e-05, l_2 2.4e-06): 6it [00:00, 58.99it/s, t=0.23, rel change=9.2e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 885.61it/s]

Solving gel with FISTA (l_1 1.7e-05, l_2 1.9e-06): 5it [00:00, 51.63it/s, t=0.24, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 921.80it/s]

Solving gel with FISTA (l_1 1.4e-05, l_2 1.5e-06): 5it [00:00, 45.85it/s, t=0.24, rel change=9e-06]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 741.70it/s]

Solving gel with FISTA (l_1 1.1e-05, l_2 1.2e-06): 3it [00:00, 22.83it/s, t=0.24, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 473.74it/s]

Solving gel with FISTA (l_1 8.6e-06, l_2 9.5e-07): 3it [00:00, 19.78it/s, t=0.25, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 679.02it/s]

Solving gel with FISTA (l_1 6.8e-06, l_2 7.5e-07): 3it [00:00, 20.75it/s, t=0.23, rel change=7.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 869.43it/s]

Solving gel with FISTA (l_1 5.4e-06, l_2 6e-07): 3it [00:00, 37.05it/s, t=0.24, rel change=7.2e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1001.97it/s]

Solving gel with FISTA (l_1 4.2e-06, l_2 4.7e-07): 2it [00:00, 26.16it/s, t=0.27, rel change=8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 957.21it/s]

Solving gel with FISTA (l_1 3.3e-06, l_2 3.7e-07): 2it [00:00, 29.20it/s, t=0.27, rel change=7.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 953.82it/s]

Solving gel with FISTA (l_1 2.6e-06, l_2 2.9e-07): 2it [00:00, 28.93it/s, t=0.27, rel change=7.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1065.19it/s]

Solving gel with FISTA (l_1 2.1e-06, l_2 2.3e-07): 2it [00:00, 26.50it/s, t=0.26, rel change=6.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 961.97it/s]

Solving gel with FISTA (l_1 0.21, l_2 0.0021): 3000it [00:16, 177.75it/s, t=6.8e-06, rel change=0.00036]
Support size: 0
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 5010.34it/s]

Solving gel with FISTA (l_1 0.17, l_2 0.0017): 6it [00:00, 189.18it/s, t=0.96, rel change=1.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3613.35it/s]

Solving gel with FISTA (l_1 0.13, l_2 0.0013): 5it [00:00, 194.12it/s, t=0.98, rel change=2e-06]  
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3217.72it/s]

Solving gel with FISTA (l_1 0.1, l_2 0.001): 23it [00:00, 120.64it/s, t=0.22, rel change=5.3e-06]
Support size: 1
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 3814.18it/s]

Solving gel with FISTA (l_1 0.082, l_2 0.00083): 33it [00:00, 135.02it/s, t=0.24, rel change=1.6e-06]
Support size: 201
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1042.09it/s]

Solving gel with FISTA (l_1 0.065, l_2 0.00065): 47it [00:00, 126.89it/s, t=0.076, rel change=7.9e-06]
Support size: 201
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 873.48it/s]

Solving gel with FISTA (l_1 0.051, l_2 0.00052): 49it [00:00, 59.15it/s, t=0.12, rel change=8.4e-06]
Support size: 221
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 682.82it/s]

Solving gel with FISTA (l_1 0.04, l_2 0.00041): 47it [00:00, 86.64it/s, t=0.051, rel change=9.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 839.95it/s]

Solving gel with FISTA (l_1 0.032, l_2 0.00032): 62it [00:00, 164.21it/s, t=0.16, rel change=9.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 765.10it/s]

Solving gel with FISTA (l_1 0.025, l_2 0.00026): 83it [00:00, 146.54it/s, t=0.026, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 963.72it/s]

Solving gel with FISTA (l_1 0.02, l_2 0.0002): 77it [00:00, 125.58it/s, t=0.016, rel change=9.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 868.90it/s]

Solving gel with FISTA (l_1 0.016, l_2 0.00016): 58it [00:00, 88.76it/s, t=0.0091, rel change=1e-05]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 903.09it/s]

Solving gel with FISTA (l_1 0.012, l_2 0.00013): 62it [00:00, 114.95it/s, t=0.094, rel change=8e-06]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 844.49it/s]

Solving gel with FISTA (l_1 0.0099, l_2 0.0001): 78it [00:00, 94.54it/s, t=0.0046, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 895.60it/s]

Solving gel with FISTA (l_1 0.0078, l_2 7.9e-05): 46it [00:00, 101.69it/s, t=0.072, rel change=1e-05] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 593.62it/s]

Solving gel with FISTA (l_1 0.0062, l_2 6.2e-05): 56it [00:00, 81.72it/s, t=0.0075, rel change=1e-05]  
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 858.02it/s]

Solving gel with FISTA (l_1 0.0049, l_2 4.9e-05): 58it [00:00, 91.43it/s, t=0.012, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 932.94it/s]

Solving gel with FISTA (l_1 0.0039, l_2 3.9e-05): 64it [00:00, 82.66it/s, t=0.02, rel change=9.6e-06] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 933.04it/s]

Solving gel with FISTA (l_1 0.003, l_2 3.1e-05): 64it [00:00, 98.91it/s, t=0.0091, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1039.09it/s]

Solving gel with FISTA (l_1 0.0024, l_2 2.4e-05): 58it [00:00, 99.34it/s, t=0.0086, rel change=9.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 904.35it/s]

Solving gel with FISTA (l_1 0.0019, l_2 1.9e-05): 46it [00:00, 94.48it/s, t=0.022, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 931.56it/s]

Solving gel with FISTA (l_1 0.0015, l_2 1.5e-05): 32it [00:00, 91.45it/s, t=0.025, rel change=9.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 838.41it/s]

Solving gel with FISTA (l_1 0.0012, l_2 1.2e-05): 44it [00:00, 91.02it/s, t=0.0092, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 897.96it/s]

Solving gel with FISTA (l_1 0.00094, l_2 9.5e-06): 40it [00:00, 96.32it/s, t=0.022, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 940.49it/s]

Solving gel with FISTA (l_1 0.00074, l_2 7.5e-06): 41it [00:00, 114.99it/s, t=0.073, rel change=9.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 856.68it/s]

Solving gel with FISTA (l_1 0.00059, l_2 5.9e-06): 28it [00:00, 83.21it/s, t=0.029, rel change=9.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 924.67it/s]

Solving gel with FISTA (l_1 0.00047, l_2 4.7e-06): 25it [00:00, 105.92it/s, t=0.17, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 800.82it/s]

Solving gel with FISTA (l_1 0.00037, l_2 3.7e-06): 17it [00:00, 90.67it/s, t=0.16, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 845.34it/s]

Solving gel with FISTA (l_1 0.00029, l_2 2.9e-06): 16it [00:00, 89.95it/s, t=0.17, rel change=8.9e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 922.51it/s]

Solving gel with FISTA (l_1 0.00023, l_2 2.3e-06): 15it [00:00, 88.87it/s, t=0.19, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 878.74it/s]

Solving gel with FISTA (l_1 0.00018, l_2 1.8e-06): 15it [00:00, 88.26it/s, t=0.19, rel change=7.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 913.84it/s]

Solving gel with FISTA (l_1 0.00014, l_2 1.5e-06): 14it [00:00, 55.58it/s, t=0.094, rel change=8.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 737.05it/s]

Solving gel with FISTA (l_1 0.00011, l_2 1.1e-06): 13it [00:00, 47.34it/s, t=0.17, rel change=8.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 707.86it/s]

Solving gel with FISTA (l_1 9e-05, l_2 9.1e-07): 13it [00:00, 77.05it/s, t=0.21, rel change=7.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 931.45it/s]

Solving gel with FISTA (l_1 7.1e-05, l_2 7.2e-07): 12it [00:00, 70.71it/s, t=0.12, rel change=7.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 960.14it/s]

Solving gel with FISTA (l_1 5.6e-05, l_2 5.7e-07): 10it [00:00, 73.81it/s, t=0.17, rel change=8.8e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 927.75it/s]

Solving gel with FISTA (l_1 4.4e-05, l_2 4.5e-07): 9it [00:00, 63.15it/s, t=0.13, rel change=8.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 957.27it/s]

Solving gel with FISTA (l_1 3.5e-05, l_2 3.5e-07): 8it [00:00, 69.36it/s, t=0.21, rel change=9.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 989.38it/s]

Solving gel with FISTA (l_1 2.8e-05, l_2 2.8e-07): 7it [00:00, 57.43it/s, t=0.2, rel change=8.9e-06] 
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 996.62it/s]

Solving gel with FISTA (l_1 2.2e-05, l_2 2.2e-07): 6it [00:00, 65.29it/s, t=0.26, rel change=9.7e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 667.49it/s]

Solving gel with FISTA (l_1 1.7e-05, l_2 1.8e-07): 5it [00:00, 33.34it/s, t=0.25, rel change=9.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 893.29it/s]

Solving gel with FISTA (l_1 1.4e-05, l_2 1.4e-07): 4it [00:00, 45.25it/s, t=0.28, rel change=9.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 932.04it/s]

Solving gel with FISTA (l_1 1.1e-05, l_2 1.1e-07): 4it [00:00, 49.81it/s, t=0.25, rel change=8.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 934.83it/s]

Solving gel with FISTA (l_1 8.6e-06, l_2 8.7e-08): 3it [00:00, 40.59it/s, t=0.26, rel change=8.5e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 373.36it/s]

Solving gel with FISTA (l_1 6.8e-06, l_2 6.8e-08): 3it [00:00, 21.43it/s, t=0.25, rel change=7.3e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 616.82it/s]

Solving gel with FISTA (l_1 5.4e-06, l_2 5.4e-08): 2it [00:00, 18.23it/s, t=0.29, rel change=8.5e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 790.40it/s]

Solving gel with FISTA (l_1 4.2e-06, l_2 4.3e-08): 2it [00:00, 27.85it/s, t=0.27, rel change=8.1e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 684.20it/s]

Solving gel with FISTA (l_1 3.3e-06, l_2 3.4e-08): 2it [00:00, 28.82it/s, t=0.27, rel change=7.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 1001.33it/s]

Solving gel with FISTA (l_1 2.6e-06, l_2 2.7e-08): 2it [00:00, 25.19it/s, t=0.27, rel change=6.6e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 765.29it/s]

Solving gel with FISTA (l_1 2.1e-06, l_2 2.1e-08): 2it [00:00, 19.36it/s, t=0.25, rel change=5.4e-06]
Support size: 226
Solving ridge regressions: 100%|██████████| 100/100 [00:00<00:00, 629.05it/s]

Let us analyze the results.





In [ ]:



    


best_summ = min(summaries.items(), key=lambda z: z[1][1])
print(best_summ)



    


















    






((0.06464223489165306, 0.0277038149535656, 0.4977026879787445), ([0, 1, 0, 1, 1, 0], 130.36752319335938))

















In [ ]:



    


best_support_hat = best_summ[1][0]
print(best_support_hat)



    


















    






[0, 1, 0, 1, 1, 0]

We were able to correctly identify the support! Now, we can perform a more thorough ridge regression on the identified support. For this we will use the ridgepaths module.





In [ ]:



    


X_tr_best_supp_hat = torch.cat(
    [A_j for A_j, s_j in zip(A_tr_stans, best_support_hat) if s_j == 1],
    dim=1,
).t()

X_val_best_supp_hat = torch.cat(
    [A_j for A_j, s_j in zip(A_val_stans, best_support_hat) if s_j == 1],
    dim=1,
).t()

def summary_best_supp_hat(support, b):
    # support is ignored
    yhat_val = (X_val_best_supp_hat.t() @ b) * y_tr_σ + y_tr_μ
    rmse = ((y_val - yhat_val) ** 2).mean().sqrt().item()
    return rmse



    











In [ ]:



    


from gel.ridgepaths import ridge_paths

l_rs_big = torch.logspace(-20, 20, 10000).tolist()

ridge_summaries = ridge_paths(
    X_tr_best_supp_hat,  # X needs to be pre-indexed using the support
    y_tr_stan,
    1,                   # this doesn't matter in this case since it is ignored; just shouldn't be None
    l_rs_big,
    summary_best_supp_hat,
    verbose=True,
)



    


















    






Solving ridge regressions: 100%|██████████| 10000/10000 [00:01<00:00, 8899.80it/s]

















In [ ]:



    


best_ridge_summ = min(ridge_summaries.items(), key=lambda z: z[1])
print(best_ridge_summ)



    


















    






(0.38190799951553345, 130.36395263671875)

The validation RMSE doesn't change much, but the identified best lambda value has changed. So now, using the identified support and ridge regularization value, we can train a final ridge regression model using the training and validation data combined to further increase performance. This can be tested using a separate test set if available.





In [ ]:



    


def dummy_summary(support, b):
    # Just return b
    return b



    











In [ ]:



    


# Combine training and validation data.
A_trvals = [torch.cat([A_j_tr, A_j_val], dim=0) for A_j_tr, A_j_val in zip(A_trs, A_vals)]
y_trval = torch.cat([y_tr, y_val])

# Compute combined statistics.
A_trval_μs = [A_j.mean(dim=0, keepdim=True) for A_j in A_trvals]
A_trval_σs = [A_j.std(dim=0, keepdim=True) for A_j in A_trvals]
y_trval_μ = y_trval.mean().item()
y_trval_σ = y_trval.std().item()

# Standardize data using combined statistics.
A_trval_stans = [(A_j - A_j_μ) / A_j_σ for A_j, A_j_μ, A_j_σ in zip(A_trvals, A_trval_μs, A_trval_σs)]
y_trval_stan = (y_trval - y_trval_μ) / y_trval_σ

# Project to support.
X_trval_best_supp_hat = torch.cat([
    A_j_trval
    for A_j_trval, s_j in zip(A_trval_stans, best_support_hat)
    if s_j == 1
], dim=1).t()
print(X_trval_best_supp_hat.shape)



    


















    






torch.Size([221, 130])

















In [ ]:



    


best_lambda = best_ridge_summ[0]
final_summary = ridge_paths(
    X_trval_best_supp_hat,
    y_trval_stan,
    1,
    [best_lambda],
    dummy_summary,
)
final_b = final_summary[best_lambda]

Here, since we know the true β, we can evaluate by comparing to it. We will assume knowledge of the fact that all features have the same normal distribution, and compute how well we estimate its parameters.





In [ ]:



    


from math import sqrt

X_trval = torch.cat(A_trvals, dim=1)
X_bias_hat = X_trval.mean().item()
X_scale_hat = X_trval.std().item()

supplen = sum(n_j for n_j, s_j in zip(ns, best_support_hat) if s_j == 1)
β_bias_hat = y_trval_μ / (X_bias_hat * supplen)
β_scale_hat = sqrt((((y_trval_σ ** 2) / supplen) - (β_bias_hat * X_scale_hat) ** 2) / ((X_bias_hat ** 2) + (X_scale_hat ** 2)))



    











In [ ]:



    


X_scale_hat, X_bias_hat  # true values (2, 3)



    


















    
Out[ ]:








(2.0057735443115234, 2.998411178588867)

















In [ ]:



    


β_scale_hat, β_bias_hat  # true values (0.125, 6)



    


















    
Out[ ]:








(1.5339424895744456, 6.003117430147738)

We are able to estimate the parameters reasonably well, with most of the error in the scale parameter. And it should be noted again, that we did perfectly recover the support. To conclude, we compute the error on the combined training validation data.





In [ ]:



    


yhat_trval = y_trval_σ * (X_trval_best_supp_hat.t() @ final_b) + y_trval_μ
rmse = ((yhat_trval - y_trval) ** 2).mean().sqrt().item()
print(rmse)



    


















    






0.4460035562515259

















In [ ]:

Content source: jayanthkoushik/torch-gel

Similar notebooks:

notebook.community | gallery | about