This script shows how one can use xcsvm. It provides shell commands and the according output. When running the example one should get similar outputs.
The following cases are shown:
As a data set we use LSHTC-small. All the examples can also be used from the file examples.bash.
In [1]:
TRAIN_DATA = "../xcsvm/xcsvm/tests/data/lshtc1_dryrun_task1_tfidf_l2_norm_applied.train.libsvm"
TEST_DATA = "../xcsvm/xcsvm/tests/data/lshtc1_dryrun_task1_tfidf_l2_norm_applied.test.libsvm"
# The following python interpreter should have xcsvm installed.
# If not used the run.py script will try to run the source code directly.
# In this case all the needed packages need to be installed.
PYTHON = "/usr/bin/python2.7"
In [2]:
! rm -rf ./model
! $PYTHON ../scripts/run.py ww_sparse --train_data $TRAIN_DATA --model_dir ./model --options folds=5 group_count=64 -v 1
[Rank 0] 2017-03-30 15:05:06,277 - base.py - INFO: Solver uses testing mode.
[Rank 0] 2017-03-30 15:05:06,334 - run.py - INFO: Loading train data.
[Rank 0] 2017-03-30 15:05:07,124 - run.py - INFO: Start training.
[Rank 0] 2017-03-30 15:05:07,124 - base.py - INFO: Use C=1 and epsilon=0.1
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:05:07,124 - base.py - INFO: Using 1 MPI processes and 1 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:05:07,152 - base.py - INFO: Remove 0 samples where K is 0.
[Rank 0] 2017-03-30 15:05:07,183 - ww.py - INFO: Use 64 groups.
[Rank 0] 2017-03-30 15:05:07,218 - base.py - INFO: Mem allocations:
[Rank 0] 2017-03-30 15:05:07,219 - base.py - INFO: Total: 522.870 MB
[Rank 0] 2017-03-30 15:05:10,192 - base.py - INFO: Iter: 1, Max. Viol.: 2.273799, Time: 0:00:02.973290
[Rank 0] 2017-03-30 15:05:13,033 - base.py - INFO: Iter: 2, Max. Viol.: 2.343154, Time: 0:00:05.814228
[Rank 0] 2017-03-30 15:05:15,865 - base.py - INFO: Iter: 3, Max. Viol.: 1.059850, Time: 0:00:08.646201
[Rank 0] 2017-03-30 15:05:18,679 - base.py - INFO: Iter: 4, Max. Viol.: 0.730385, Time: 0:00:11.459940
[Rank 0] 2017-03-30 15:05:21,478 - base.py - INFO: Iter: 5, Max. Viol.: 0.381688, Time: 0:00:14.259243
[Rank 0] 2017-03-30 15:05:24,262 - base.py - INFO: Iter: 6, Max. Viol.: 0.277613, Time: 0:00:17.042925
[Rank 0] 2017-03-30 15:05:27,043 - base.py - INFO: Iter: 7, Max. Viol.: 0.249915, Time: 0:00:19.824264
[Rank 0] 2017-03-30 15:05:29,809 - base.py - INFO: Iter: 8, Max. Viol.: 0.243094, Time: 0:00:22.590356
[Rank 0] 2017-03-30 15:05:32,575 - base.py - INFO: Iter: 9, Max. Viol.: 0.191051, Time: 0:00:25.356007
[Rank 0] 2017-03-30 15:05:35,337 - base.py - INFO: Iter: 10, Max. Viol.: 0.150705, Time: 0:00:28.117966
[Rank 0] 2017-03-30 15:06:01,239 - base.py - INFO: Iter: 19, Max. Viol.: 0.000000, Time: 0:00:54.020156
[Rank 0] 2017-03-30 15:06:01,239 - base.py - INFO: Optimization stopped after 19 iterations.
[Rank 0] 2017-03-30 15:06:01,368 - base.py - INFO: Sparsity of W is: 0.0594432
[Rank 0] 2017-03-30 15:06:01,368 - base.py - INFO: Sparsity and sum of alpha are: 0.0109703 - 4607.21
[Rank 0] 2017-03-30 15:06:01,896 - run.py - INFO: Training time: 54.772 s
[Rank 0] 2017-03-30 15:06:01,896 - run.py - INFO: Storing model.
[Rank 0] 2017-03-30 15:06:02,449 - run.py - INFO: Serializing time: 0.553 s
In [3]:
! $PYTHON ../scripts/run.py ww_sparse --test_data $TEST_DATA --model_dir ./model
[Rank 0] 2017-03-30 15:06:45,414 - run.py - INFO: Loading model.
[Rank 0] 2017-03-30 15:06:45,414 - base.py - INFO: Solver uses testing mode.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:06:45,469 - base.py - INFO: Using 1 MPI processes and 1 MP threads.
[Rank 0] 2017-03-30 15:06:45,906 - run.py - INFO: Unserializing time: 0.491 s
[Rank 0] 2017-03-30 15:06:45,906 - run.py - INFO: Loading test data.
[Rank 0] 2017-03-30 15:06:46,226 - run.py - INFO: Start prediction.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:06:46,226 - base.py - INFO: Using 1 MPI processes and 1 MP threads.
[Rank 0] 2017-03-30 15:06:46,852 - run.py - INFO: Test time: 0.626 s
[Rank 0] 2017-03-30 15:06:46,852 - run.py - INFO: Accuracy: 0.452099
In [4]:
! $PYTHON ../scripts/run.py ww_sparse --train_data $TRAIN_DATA --test_data $TEST_DATA --options folds=5 group_count=64 -v 1
[Rank 0] 2017-03-30 15:06:51,488 - base.py - INFO: Solver uses testing mode.
[Rank 0] 2017-03-30 15:06:51,542 - run.py - INFO: Loading train data.
[Rank 0] 2017-03-30 15:06:52,353 - run.py - INFO: Start training.
[Rank 0] 2017-03-30 15:06:52,353 - base.py - INFO: Use C=1 and epsilon=0.1
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:06:52,353 - base.py - INFO: Using 1 MPI processes and 1 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:06:52,378 - base.py - INFO: Remove 0 samples where K is 0.
[Rank 0] 2017-03-30 15:06:52,413 - ww.py - INFO: Use 64 groups.
[Rank 0] 2017-03-30 15:06:52,449 - base.py - INFO: Mem allocations:
[Rank 0] 2017-03-30 15:06:52,450 - base.py - INFO: Total: 522.870 MB
[Rank 0] 2017-03-30 15:06:55,577 - base.py - INFO: Iter: 1, Max. Viol.: 2.273799, Time: 0:00:03.127347
[Rank 0] 2017-03-30 15:06:59,072 - base.py - INFO: Iter: 2, Max. Viol.: 2.343154, Time: 0:00:06.621902
[Rank 0] 2017-03-30 15:07:01,963 - base.py - INFO: Iter: 3, Max. Viol.: 1.059850, Time: 0:00:09.512923
[Rank 0] 2017-03-30 15:07:04,788 - base.py - INFO: Iter: 4, Max. Viol.: 0.730385, Time: 0:00:12.337820
[Rank 0] 2017-03-30 15:07:07,719 - base.py - INFO: Iter: 5, Max. Viol.: 0.381688, Time: 0:00:15.268811
[Rank 0] 2017-03-30 15:07:10,634 - base.py - INFO: Iter: 6, Max. Viol.: 0.277613, Time: 0:00:18.183486
[Rank 0] 2017-03-30 15:07:13,429 - base.py - INFO: Iter: 7, Max. Viol.: 0.249915, Time: 0:00:20.978478
[Rank 0] 2017-03-30 15:07:16,213 - base.py - INFO: Iter: 8, Max. Viol.: 0.243094, Time: 0:00:23.763388
[Rank 0] 2017-03-30 15:07:18,995 - base.py - INFO: Iter: 9, Max. Viol.: 0.191051, Time: 0:00:26.544906
[Rank 0] 2017-03-30 15:07:21,792 - base.py - INFO: Iter: 10, Max. Viol.: 0.150705, Time: 0:00:29.342025
[Rank 0] 2017-03-30 15:07:47,349 - base.py - INFO: Iter: 19, Max. Viol.: 0.000000, Time: 0:00:54.898683
[Rank 0] 2017-03-30 15:07:47,349 - base.py - INFO: Optimization stopped after 19 iterations.
[Rank 0] 2017-03-30 15:07:47,508 - base.py - INFO: Sparsity of W is: 0.0594432
[Rank 0] 2017-03-30 15:07:47,509 - base.py - INFO: Sparsity and sum of alpha are: 0.0109703 - 4607.21
[Rank 0] 2017-03-30 15:07:48,189 - run.py - INFO: Training time: 55.836 s
[Rank 0] 2017-03-30 15:07:48,189 - run.py - INFO: Loading test data.
[Rank 0] 2017-03-30 15:07:48,717 - run.py - INFO: Start prediction.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:07:48,718 - base.py - INFO: Using 1 MPI processes and 1 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:07:49,423 - run.py - INFO: Test time: 0.705 s
[Rank 0] 2017-03-30 15:07:49,423 - run.py - INFO: Accuracy: 0.452099
In [5]:
! $PYTHON ../scripts/run.py ww_sparse --train_data $TRAIN_DATA --test_data $TEST_DATA --options nr_threads=2 folds=5 group_count=64 -v 1
[Rank 0] 2017-03-30 15:08:39,709 - base.py - INFO: Solver uses testing mode.
[Rank 0] 2017-03-30 15:08:39,768 - run.py - INFO: Loading train data.
[Rank 0] 2017-03-30 15:08:40,543 - run.py - INFO: Start training.
[Rank 0] 2017-03-30 15:08:40,543 - base.py - INFO: Use C=1 and epsilon=0.1
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:08:40,543 - base.py - INFO: Using 1 MPI processes and 2 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:08:40,568 - base.py - INFO: Remove 0 samples where K is 0.
[Rank 0] 2017-03-30 15:08:40,597 - ww.py - INFO: Use 64 groups.
[Rank 0] 2017-03-30 15:08:40,632 - base.py - INFO: Mem allocations:
[Rank 0] 2017-03-30 15:08:40,632 - base.py - INFO: Total: 522.870 MB
[Rank 0] 2017-03-30 15:08:42,376 - base.py - INFO: Iter: 1, Max. Viol.: 2.273799, Time: 0:00:01.743913
[Rank 0] 2017-03-30 15:08:44,061 - base.py - INFO: Iter: 2, Max. Viol.: 2.343154, Time: 0:00:03.428325
[Rank 0] 2017-03-30 15:08:45,768 - base.py - INFO: Iter: 3, Max. Viol.: 1.059850, Time: 0:00:05.135508
[Rank 0] 2017-03-30 15:08:47,451 - base.py - INFO: Iter: 4, Max. Viol.: 0.730385, Time: 0:00:06.818456
[Rank 0] 2017-03-30 15:08:49,139 - base.py - INFO: Iter: 5, Max. Viol.: 0.381688, Time: 0:00:08.506632
[Rank 0] 2017-03-30 15:08:50,786 - base.py - INFO: Iter: 6, Max. Viol.: 0.277613, Time: 0:00:10.153579
[Rank 0] 2017-03-30 15:08:52,428 - base.py - INFO: Iter: 7, Max. Viol.: 0.249915, Time: 0:00:11.795652
[Rank 0] 2017-03-30 15:08:54,102 - base.py - INFO: Iter: 8, Max. Viol.: 0.243094, Time: 0:00:13.469609
[Rank 0] 2017-03-30 15:08:55,827 - base.py - INFO: Iter: 9, Max. Viol.: 0.191051, Time: 0:00:15.194482
[Rank 0] 2017-03-30 15:08:57,544 - base.py - INFO: Iter: 10, Max. Viol.: 0.150705, Time: 0:00:16.911303
[Rank 0] 2017-03-30 15:09:14,030 - base.py - INFO: Iter: 19, Max. Viol.: 0.000000, Time: 0:00:33.397958
[Rank 0] 2017-03-30 15:09:14,031 - base.py - INFO: Optimization stopped after 19 iterations.
[Rank 0] 2017-03-30 15:09:14,170 - base.py - INFO: Sparsity of W is: 0.0594432
[Rank 0] 2017-03-30 15:09:14,170 - base.py - INFO: Sparsity and sum of alpha are: 0.0109703 - 4607.21
[Rank 0] 2017-03-30 15:09:14,708 - run.py - INFO: Training time: 34.166 s
[Rank 0] 2017-03-30 15:09:14,709 - run.py - INFO: Loading test data.
[Rank 0] 2017-03-30 15:09:15,091 - run.py - INFO: Start prediction.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:09:15,091 - base.py - INFO: Using 1 MPI processes and 2 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:09:15,747 - run.py - INFO: Test time: 0.656 s
[Rank 0] 2017-03-30 15:09:15,748 - run.py - INFO: Accuracy: 0.452099
In [6]:
! mpiexec -n 2 $PYTHON ../scripts/run.py ww_sparse --train_data $TRAIN_DATA --test_data $TEST_DATA --options folds=5 group_count=64 -v 1
[Rank 0] 2017-03-30 15:09:34,291 - base.py - INFO: Solver uses testing mode.
[Rank 0] 2017-03-30 15:09:34,352 - run.py - INFO: Loading train data.
[Rank 0] 2017-03-30 15:09:35,235 - run.py - INFO: Start training.
[Rank 0] 2017-03-30 15:09:35,235 - base.py - INFO: Use C=1 and epsilon=0.1
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:09:35,235 - base.py - INFO: Using 2 MPI processes and 1 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:09:35,364 - base.py - INFO: Remove 0 samples where K is 0.
[Rank 0] 2017-03-30 15:09:35,402 - ww.py - INFO: Use 64 groups.
[Rank 0] 2017-03-30 15:09:35,454 - base.py - INFO: Mem allocations:
[Rank 0] 2017-03-30 15:09:35,454 - base.py - INFO: Total: 401.959 MB
[Rank 0] 2017-03-30 15:09:37,268 - base.py - INFO: Iter: 1, Max. Viol.: 1.860290, Time: 0:00:01.806321
[Rank 0] 2017-03-30 15:09:39,018 - base.py - INFO: Iter: 2, Max. Viol.: 1.865952, Time: 0:00:03.556403
[Rank 0] 2017-03-30 15:09:40,835 - base.py - INFO: Iter: 3, Max. Viol.: 0.812660, Time: 0:00:05.372441
[Rank 0] 2017-03-30 15:09:42,691 - base.py - INFO: Iter: 4, Max. Viol.: 0.449393, Time: 0:00:07.228644
[Rank 0] 2017-03-30 15:09:44,549 - base.py - INFO: Iter: 5, Max. Viol.: 0.326411, Time: 0:00:09.087122
[Rank 0] 2017-03-30 15:09:46,389 - base.py - INFO: Iter: 6, Max. Viol.: 0.202817, Time: 0:00:10.927235
[Rank 0] 2017-03-30 15:09:48,254 - base.py - INFO: Iter: 7, Max. Viol.: 0.266039, Time: 0:00:12.791925
[Rank 0] 2017-03-30 15:09:50,064 - base.py - INFO: Iter: 8, Max. Viol.: 0.164717, Time: 0:00:14.601591
[Rank 0] 2017-03-30 15:09:51,908 - base.py - INFO: Iter: 9, Max. Viol.: 0.158098, Time: 0:00:16.446103
[Rank 0] 2017-03-30 15:09:54,261 - base.py - INFO: Iter: 10, Max. Viol.: 0.146283, Time: 0:00:18.798770
[Rank 0] 2017-03-30 15:10:10,778 - base.py - INFO: Iter: 17, Max. Viol.: 0.000000, Time: 0:00:35.315510
[Rank 0] 2017-03-30 15:10:10,778 - base.py - INFO: Optimization stopped after 17 iterations.
[Rank 0] 2017-03-30 15:10:10,968 - base.py - INFO: Sparsity of W is: 0.0982164
[Rank 0] 2017-03-30 15:10:10,968 - base.py - INFO: Sparsity and sum of alpha are: 0.0109878 - 4601.15
[Rank 0] 2017-03-30 15:10:11,519 - run.py - INFO: Training time: 36.284 s
[Rank 0] 2017-03-30 15:10:11,519 - run.py - INFO: Loading test data.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:10:12,334 - run.py - INFO: Start prediction.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.base <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.base' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/base.so'>
[Rank 0] 2017-03-30 15:10:12,334 - base.py - INFO: Using 2 MPI processes and 1 MP threads.
<type 'numpy.float64'> <type 'numpy.uint64'> .dtype_f64_idtype_ui64.ww <module 'xcsvm.solvers.cython.dtype_f64_idtype_ui64.ww' from '/home/alber/.pyxbld_testing/lib.linux-x86_64-2.7/xcsvm/solvers/cython/dtype_f64_idtype_ui64/ww.so'>
[Rank 0] 2017-03-30 15:10:12,949 - run.py - INFO: Test time: 0.615 s
[Rank 0] 2017-03-30 15:10:12,949 - run.py - INFO: Accuracy: 0.452637
Content source: albermax/xcsvm
Similar notebooks: