In [2]:
time echo {1..5} | xargs -n 1 -P 5 sleep
real 0m5.005s
user 0m0.000s
sys 0m0.005s
In [6]:
xargs --help
Usage: xargs [OPTION]... COMMAND [INITIAL-ARGS]...
Run COMMAND with arguments INITIAL-ARGS and more arguments read from input.
Mandatory and optional arguments to long options are also
mandatory or optional for the corresponding short option.
-0, --null items are separated by a null, not whitespace;
disables quote and backslash processing and
logical EOF processing
-a, --arg-file=FILE read arguments from FILE, not standard input
-d, --delimiter=CHARACTER items in input stream are separated by CHARACTER,
not by whitespace; disables quote and backslash
processing and logical EOF processing
-E END set logical EOF string; if END occurs as a line
of input, the rest of the input is ignored
(ignored if -0 or -d was specified)
-e, --eof[=END] equivalent to -E END if END is specified;
otherwise, there is no end-of-file string
-I R same as --replace=R
-i, --replace[=R] replace R in INITIAL-ARGS with names read
from standard input; if R is unspecified,
assume {}
-L, --max-lines=MAX-LINES use at most MAX-LINES non-blank input lines per
command line
-l[MAX-LINES] similar to -L but defaults to at most one non-
blank input line if MAX-LINES is not specified
-n, --max-args=MAX-ARGS use at most MAX-ARGS arguments per command line
-P, --max-procs=MAX-PROCS run at most MAX-PROCS processes at a time
-p, --interactive prompt before running commands
--process-slot-var=VAR set environment variable VAR in child processes
-r, --no-run-if-empty if there are no arguments, then do not run COMMAND;
if this option is not given, COMMAND will be
run at least once
-s, --max-chars=MAX-CHARS limit length of command line to MAX-CHARS
--show-limits show limits on command-line length
-t, --verbose print commands before executing them
-x, --exit exit if the size (see -s) is exceeded
--help display this help and exit
--version output version information and exit
Report bugs to <bug-findutils@gnu.org>.
In [10]:
time echo {1..5} | xargs -n 1 -P 5 echo
#-n 1 每次只读入一个参数
# -P 5 分配5个CPU
1
2
3
4
5
real 0m0.009s
user 0m0.001s
sys 0m0.007s
In [30]:
time seq 77 80 | parallel wget http://www.gutenberg.org/cache/epub/{}/pg{}.txt
#并行下载文件
#其中{}是占位符,用来占位传入参数的位置。
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; and it won't cost you a cent.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence the citation notice: run 'parallel --bibtex'.
--2017-01-20 20:27:26-- http://www.gutenberg.org/cache/epub/79/pg79.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:27:27-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg79.txt.3'
0K ......... 100% 3.18M=0.003s
2017-01-20 20:27:28 (3.18 MB/s) - 'pg79.txt.3' saved [9527/9527]
--2017-01-20 20:27:26-- http://www.gutenberg.org/cache/epub/77/pg77.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:27:27-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg77.txt.3'
0K ......... 100% 2.14M=0.004s
2017-01-20 20:27:28 (2.14 MB/s) - 'pg77.txt.3' saved [9527/9527]
--2017-01-20 20:27:26-- http://www.gutenberg.org/cache/epub/78/pg78.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:27:27-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg78.txt.3'
0K ......... 100% 1.20M=0.008s
2017-01-20 20:27:28 (1.20 MB/s) - 'pg78.txt.3' saved [9527/9527]
--2017-01-20 20:27:26-- http://www.gutenberg.org/cache/epub/80/pg80.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:27:27-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg80.txt.3'
0K ......... 100% 23.8K=0.4s
2017-01-20 20:27:28 (23.8 KB/s) - 'pg80.txt.3' saved [9527/9527]
real 0m2.332s
user 0m0.189s
sys 0m0.041s
In [29]:
time seq 77 80 | awk '{print "http://www.gutenberg.org/cache/epub/"$1"/pg"$1".txt"}' | parallel wget
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; and it won't cost you a cent.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence the citation notice: run 'parallel --bibtex'.
--2017-01-20 20:26:53-- http://www.gutenberg.org/cache/epub/80/pg80.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:26:54-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9534 (9.3K) [text/html]
Saving to: 'pg80.txt.2'
0K ......... 100% 1.44M=0.006s
2017-01-20 20:26:55 (1.44 MB/s) - 'pg80.txt.2' saved [9534/9534]
--2017-01-20 20:26:53-- http://www.gutenberg.org/cache/epub/78/pg78.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:26:54-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg78.txt.2'
0K ......... 100% 1.13M=0.008s
2017-01-20 20:26:55 (1.13 MB/s) - 'pg78.txt.2' saved [9527/9527]
--2017-01-20 20:26:53-- http://www.gutenberg.org/cache/epub/79/pg79.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: http://www.gutenberg.org/w/captcha/question/ [following]
--2017-01-20 20:26:54-- http://www.gutenberg.org/w/captcha/question/
Reusing existing connection to www.gutenberg.org:80.
HTTP request sent, awaiting response... 200 OK
Length: 9527 (9.3K) [text/html]
Saving to: 'pg79.txt.2'
0K ......... 100% 849K=0.01s
2017-01-20 20:26:55 (849 KB/s) - 'pg79.txt.2' saved [9527/9527]
--2017-01-20 20:26:53-- http://www.gutenberg.org/cache/epub/77/pg77.txt
Resolving www.gutenberg.org... 152.19.134.47, 2610:28:3090:3000:0:bad:cafe:47
Connecting to www.gutenberg.org|152.19.134.47|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 646211 (631K) [text/plain]
Saving to: 'pg77.txt.2'
0K .......... .......... .......... .......... .......... 7% 28.8K 20s
50K .......... .......... .......... .......... .......... 15% 44.3K 15s
100K .......... .......... .......... .......... .......... 23% 51.0K 12s
150K .......... .......... .......... .......... .......... 31% 39.2K 11s
200K .......... .......... .......... .......... .......... 39% 81.3K 9s
250K .......... .......... .......... .......... .......... 47% 81.5K 7s
300K .......... .......... .......... .......... .......... 55% 158K 5s
350K .......... .......... .......... .......... .......... 63% 162K 4s
400K .......... .......... .......... .......... .......... 71% 161K 3s
450K .......... .......... .......... .......... .......... 79% 78.1K 2s
500K .......... .......... .......... .......... .......... 87% 152K 1s
550K .......... .......... .......... .......... .......... 95% 91.6K 0s
600K .......... .......... .......... . 100% 1.77M=8.8s
2017-01-20 20:27:03 (71.6 KB/s) - 'pg77.txt.2' saved [646211/646211]
real 0m9.977s
user 0m0.192s
sys 0m0.069s
In [31]:
time for x in `cat download_link.txt`; do echo $x; done
http://www.gutenberg.org/cache/epub/1/pg1.txt
http://www.gutenberg.org/cache/epub/2/pg2.txt
http://www.gutenberg.org/cache/epub/3/pg3.txt
http://www.gutenberg.org/cache/epub/4/pg4.txt
http://www.gutenberg.org/cache/epub/5/pg5.txt
http://www.gutenberg.org/cache/epub/6/pg6.txt
http://www.gutenberg.org/cache/epub/7/pg7.txt
http://www.gutenberg.org/cache/epub/8/pg8.txt
http://www.gutenberg.org/cache/epub/9/pg9.txt
http://www.gutenberg.org/cache/epub/10/pg10.txt
http://www.gutenberg.org/cache/epub/11/pg11.txt
http://www.gutenberg.org/cache/epub/12/pg12.txt
http://www.gutenberg.org/cache/epub/13/pg13.txt
http://www.gutenberg.org/cache/epub/14/pg14.txt
http://www.gutenberg.org/cache/epub/15/pg15.txt
http://www.gutenberg.org/cache/epub/16/pg16.txt
http://www.gutenberg.org/cache/epub/17/pg17.txt
http://www.gutenberg.org/cache/epub/18/pg18.txt
http://www.gutenberg.org/cache/epub/19/pg19.txt
http://www.gutenberg.org/cache/epub/20/pg20.txt
http://www.gutenberg.org/cache/epub/21/pg21.txt
http://www.gutenberg.org/cache/epub/22/pg22.txt
http://www.gutenberg.org/cache/epub/23/pg23.txt
http://www.gutenberg.org/cache/epub/24/pg24.txt
http://www.gutenberg.org/cache/epub/25/pg25.txt
http://www.gutenberg.org/cache/epub/26/pg26.txt
http://www.gutenberg.org/cache/epub/27/pg27.txt
http://www.gutenberg.org/cache/epub/28/pg28.txt
http://www.gutenberg.org/cache/epub/29/pg29.txt
http://www.gutenberg.org/cache/epub/30/pg30.txt
http://www.gutenberg.org/cache/epub/31/pg31.txt
http://www.gutenberg.org/cache/epub/32/pg32.txt
http://www.gutenberg.org/cache/epub/33/pg33.txt
http://www.gutenberg.org/cache/epub/34/pg34.txt
http://www.gutenberg.org/cache/epub/35/pg35.txt
http://www.gutenberg.org/cache/epub/36/pg36.txt
http://www.gutenberg.org/cache/epub/37/pg37.txt
http://www.gutenberg.org/cache/epub/38/pg38.txt
http://www.gutenberg.org/cache/epub/39/pg39.txt
http://www.gutenberg.org/cache/epub/40/pg40.txt
http://www.gutenberg.org/cache/epub/41/pg41.txt
http://www.gutenberg.org/cache/epub/42/pg42.txt
http://www.gutenberg.org/cache/epub/43/pg43.txt
http://www.gutenberg.org/cache/epub/44/pg44.txt
http://www.gutenberg.org/cache/epub/45/pg45.txt
http://www.gutenberg.org/cache/epub/46/pg46.txt
http://www.gutenberg.org/cache/epub/47/pg47.txt
http://www.gutenberg.org/cache/epub/48/pg48.txt
http://www.gutenberg.org/cache/epub/49/pg49.txt
http://www.gutenberg.org/cache/epub/50/pg50.txt
http://www.gutenberg.org/cache/epub/51/pg51.txt
http://www.gutenberg.org/cache/epub/52/pg52.txt
http://www.gutenberg.org/cache/epub/53/pg53.txt
http://www.gutenberg.org/cache/epub/54/pg54.txt
http://www.gutenberg.org/cache/epub/55/pg55.txt
http://www.gutenberg.org/cache/epub/56/pg56.txt
http://www.gutenberg.org/cache/epub/57/pg57.txt
http://www.gutenberg.org/cache/epub/58/pg58.txt
http://www.gutenberg.org/cache/epub/59/pg59.txt
http://www.gutenberg.org/cache/epub/60/pg60.txt
http://www.gutenberg.org/cache/epub/61/pg61.txt
http://www.gutenberg.org/cache/epub/62/pg62.txt
http://www.gutenberg.org/cache/epub/63/pg63.txt
http://www.gutenberg.org/cache/epub/64/pg64.txt
http://www.gutenberg.org/cache/epub/65/pg65.txt
http://www.gutenberg.org/cache/epub/66/pg66.txt
http://www.gutenberg.org/cache/epub/67/pg67.txt
http://www.gutenberg.org/cache/epub/68/pg68.txt
http://www.gutenberg.org/cache/epub/69/pg69.txt
http://www.gutenberg.org/cache/epub/70/pg70.txt
http://www.gutenberg.org/cache/epub/71/pg71.txt
http://www.gutenberg.org/cache/epub/72/pg72.txt
http://www.gutenberg.org/cache/epub/73/pg73.txt
http://www.gutenberg.org/cache/epub/74/pg74.txt
http://www.gutenberg.org/cache/epub/75/pg75.txt
http://www.gutenberg.org/cache/epub/76/pg76.txt
http://www.gutenberg.org/cache/epub/77/pg77.txt
http://www.gutenberg.org/cache/epub/78/pg78.txt
http://www.gutenberg.org/cache/epub/79/pg79.txt
http://www.gutenberg.org/cache/epub/80/pg80.txt
http://www.gutenberg.org/cache/epub/81/pg81.txt
http://www.gutenberg.org/cache/epub/82/pg82.txt
http://www.gutenberg.org/cache/epub/83/pg83.txt
http://www.gutenberg.org/cache/epub/84/pg84.txt
http://www.gutenberg.org/cache/epub/85/pg85.txt
http://www.gutenberg.org/cache/epub/86/pg86.txt
http://www.gutenberg.org/cache/epub/87/pg87.txt
http://www.gutenberg.org/cache/epub/88/pg88.txt
http://www.gutenberg.org/cache/epub/89/pg89.txt
http://www.gutenberg.org/cache/epub/90/pg90.txt
http://www.gutenberg.org/cache/epub/91/pg91.txt
http://www.gutenberg.org/cache/epub/92/pg92.txt
http://www.gutenberg.org/cache/epub/93/pg93.txt
http://www.gutenberg.org/cache/epub/94/pg94.txt
http://www.gutenberg.org/cache/epub/95/pg95.txt
http://www.gutenberg.org/cache/epub/96/pg96.txt
http://www.gutenberg.org/cache/epub/97/pg97.txt
http://www.gutenberg.org/cache/epub/98/pg98.txt
http://www.gutenberg.org/cache/epub/99/pg99.txt
http://www.gutenberg.org/cache/epub/100/pg100.txt
real 0m0.003s
user 0m0.002s
sys 0m0.001s
In [33]:
time cat download_link.txt | parallel echo {}
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; and it won't cost you a cent.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence the citation notice: run 'parallel --bibtex'.
http://www.gutenberg.org/cache/epub/1/pg1.txt
http://www.gutenberg.org/cache/epub/2/pg2.txt
http://www.gutenberg.org/cache/epub/3/pg3.txt
http://www.gutenberg.org/cache/epub/4/pg4.txt
http://www.gutenberg.org/cache/epub/5/pg5.txt
http://www.gutenberg.org/cache/epub/6/pg6.txt
http://www.gutenberg.org/cache/epub/7/pg7.txt
http://www.gutenberg.org/cache/epub/8/pg8.txt
http://www.gutenberg.org/cache/epub/9/pg9.txt
http://www.gutenberg.org/cache/epub/10/pg10.txt
http://www.gutenberg.org/cache/epub/11/pg11.txt
http://www.gutenberg.org/cache/epub/12/pg12.txt
http://www.gutenberg.org/cache/epub/13/pg13.txt
http://www.gutenberg.org/cache/epub/14/pg14.txt
http://www.gutenberg.org/cache/epub/15/pg15.txt
http://www.gutenberg.org/cache/epub/16/pg16.txt
http://www.gutenberg.org/cache/epub/17/pg17.txt
http://www.gutenberg.org/cache/epub/18/pg18.txt
http://www.gutenberg.org/cache/epub/19/pg19.txt
http://www.gutenberg.org/cache/epub/20/pg20.txt
http://www.gutenberg.org/cache/epub/21/pg21.txt
http://www.gutenberg.org/cache/epub/22/pg22.txt
http://www.gutenberg.org/cache/epub/23/pg23.txt
http://www.gutenberg.org/cache/epub/24/pg24.txt
http://www.gutenberg.org/cache/epub/25/pg25.txt
http://www.gutenberg.org/cache/epub/26/pg26.txt
http://www.gutenberg.org/cache/epub/27/pg27.txt
http://www.gutenberg.org/cache/epub/28/pg28.txt
http://www.gutenberg.org/cache/epub/29/pg29.txt
http://www.gutenberg.org/cache/epub/30/pg30.txt
http://www.gutenberg.org/cache/epub/31/pg31.txt
http://www.gutenberg.org/cache/epub/32/pg32.txt
http://www.gutenberg.org/cache/epub/33/pg33.txt
http://www.gutenberg.org/cache/epub/34/pg34.txt
http://www.gutenberg.org/cache/epub/35/pg35.txt
http://www.gutenberg.org/cache/epub/36/pg36.txt
http://www.gutenberg.org/cache/epub/37/pg37.txt
http://www.gutenberg.org/cache/epub/38/pg38.txt
http://www.gutenberg.org/cache/epub/39/pg39.txt
http://www.gutenberg.org/cache/epub/40/pg40.txt
http://www.gutenberg.org/cache/epub/41/pg41.txt
http://www.gutenberg.org/cache/epub/42/pg42.txt
http://www.gutenberg.org/cache/epub/43/pg43.txt
http://www.gutenberg.org/cache/epub/44/pg44.txt
http://www.gutenberg.org/cache/epub/45/pg45.txt
http://www.gutenberg.org/cache/epub/46/pg46.txt
http://www.gutenberg.org/cache/epub/47/pg47.txt
http://www.gutenberg.org/cache/epub/48/pg48.txt
http://www.gutenberg.org/cache/epub/49/pg49.txt
http://www.gutenberg.org/cache/epub/50/pg50.txt
http://www.gutenberg.org/cache/epub/51/pg51.txt
http://www.gutenberg.org/cache/epub/52/pg52.txt
http://www.gutenberg.org/cache/epub/53/pg53.txt
http://www.gutenberg.org/cache/epub/54/pg54.txt
http://www.gutenberg.org/cache/epub/55/pg55.txt
http://www.gutenberg.org/cache/epub/56/pg56.txt
http://www.gutenberg.org/cache/epub/57/pg57.txt
http://www.gutenberg.org/cache/epub/58/pg58.txt
http://www.gutenberg.org/cache/epub/59/pg59.txt
http://www.gutenberg.org/cache/epub/60/pg60.txt
http://www.gutenberg.org/cache/epub/61/pg61.txt
http://www.gutenberg.org/cache/epub/62/pg62.txt
http://www.gutenberg.org/cache/epub/63/pg63.txt
http://www.gutenberg.org/cache/epub/64/pg64.txt
http://www.gutenberg.org/cache/epub/65/pg65.txt
http://www.gutenberg.org/cache/epub/66/pg66.txt
http://www.gutenberg.org/cache/epub/67/pg67.txt
http://www.gutenberg.org/cache/epub/68/pg68.txt
http://www.gutenberg.org/cache/epub/69/pg69.txt
http://www.gutenberg.org/cache/epub/70/pg70.txt
http://www.gutenberg.org/cache/epub/71/pg71.txt
http://www.gutenberg.org/cache/epub/72/pg72.txt
http://www.gutenberg.org/cache/epub/73/pg73.txt
http://www.gutenberg.org/cache/epub/74/pg74.txt
http://www.gutenberg.org/cache/epub/75/pg75.txt
http://www.gutenberg.org/cache/epub/76/pg76.txt
http://www.gutenberg.org/cache/epub/77/pg77.txt
http://www.gutenberg.org/cache/epub/78/pg78.txt
http://www.gutenberg.org/cache/epub/79/pg79.txt
http://www.gutenberg.org/cache/epub/80/pg80.txt
http://www.gutenberg.org/cache/epub/81/pg81.txt
http://www.gutenberg.org/cache/epub/82/pg82.txt
http://www.gutenberg.org/cache/epub/83/pg83.txt
http://www.gutenberg.org/cache/epub/84/pg84.txt
http://www.gutenberg.org/cache/epub/85/pg85.txt
http://www.gutenberg.org/cache/epub/86/pg86.txt
http://www.gutenberg.org/cache/epub/87/pg87.txt
http://www.gutenberg.org/cache/epub/88/pg88.txt
http://www.gutenberg.org/cache/epub/89/pg89.txt
http://www.gutenberg.org/cache/epub/90/pg90.txt
http://www.gutenberg.org/cache/epub/91/pg91.txt
http://www.gutenberg.org/cache/epub/92/pg92.txt
http://www.gutenberg.org/cache/epub/93/pg93.txt
http://www.gutenberg.org/cache/epub/94/pg94.txt
http://www.gutenberg.org/cache/epub/95/pg95.txt
http://www.gutenberg.org/cache/epub/96/pg96.txt
http://www.gutenberg.org/cache/epub/97/pg97.txt
http://www.gutenberg.org/cache/epub/98/pg98.txt
http://www.gutenberg.org/cache/epub/99/pg99.txt
http://www.gutenberg.org/cache/epub/100/pg100.txt
real 0m0.537s
user 0m0.397s
sys 0m0.183s
In [ ]:
Content source: xianjunzhengbackup/code
Similar notebooks: