15 wget command examples

Wget is to download download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.


Syntax:

wget [option] … [URL] …

Multiple options can be used, for example

wget -rpk http://www.example.com

Usage Examples:

Download file

wget http://www.example.com/awesome_pr0n.html

Download a file whose URL contains a “&” (note quotes)

wget “http://www.example.com/min.php?page=home&user=ma”

Download, in addition to the specified file, all files required for optimal viewing

wget -p http://www.example.com/index.html

Draw an entire site (recursive download)

wget -r http://www.example.com/

Resume a download

wget -c http://www.example.com/xyz.html

Specify the name of the file once downloaded

wget --output-document = index.html http://www.example.com/index.php?page=home

Download files only an extension (here,. Jpg)

wget -A .jpg http://www.example.com/

Limit download speed

wget --limit-rate = 30k http://www.example.com/

Using a file containing addresses for download

wget -i liste_de_zyx.txt

Mp3 download all addresses contained in this list

wget -r-l1-H-t1-nd-N-np-a .mp3 erobots = off-imp3_sites.txt

Links automatically convert to a local consultation pages

wget -k http://www.example.com/

Download via ftp (With authentication)

wget -r l4 ftp://username:password @ example.com /

Create a backup of your del.icio.us bookmarks

wget http://del.icio.us/username/

Use wget to display the page source on the terminal

wget -qO – http://www.example.com/blog

For instance, if your .wgetrc
sets “exclude_directories” to /cgi-bin, the following example will
first reset it, and then set it to exclude /~nobody and /~somebody.
You can also clear the lists in .wgetrc.

wget -X ” -X /~nobody,/~somebody

Check out the list of Windows' failed rivals. Its longer than you think. Click here.
C++ hating: An article. Click here.
To subscribe to the "Guy WhoSteals" feed, click here.
You can add yourself to the GuyWhoSteals fanpage on Facebook or follow GuyWhoSteals on Twitter.

The timeless utility of sed and awk (A little knowledge of CLI goes a long way)

Now over 30 years old, the UNIX command line utilities sed and awk are useful tools for cleaning up and manipulating data. In their Taxonomy of Data Science, Hilary Mason and Chris Wiggins note that when cleaning data, "Sed, awk, grep are enough for most small tasks, and using either Perl or Python should be good enough for the rest." A little aptitude with command line tools can go a long way.

sed is a stream editor: it operates on data in a serial fashion as it reads it. You can think of sed as a way to batch up a bunch of search and replace operations that you might perform in a text editor. For instance, this command will replace all instances of "foo" with "bar" within a file:

sed -e 's/foo/bar/g' myfile.txt

Anybody who has used regular expressions within a text editor or 
programming language
will find sed easy to grasp. Awk takes a 
little more getting used
to. A record-oriented tool, awk is the right tool to use when 
your data contains
delimited fields that you want to manipulate.

Consider this list of names, which we'll imagine lives in the file presidents.txt.

George Washington
John Adams
Thomas Jefferson
James Madison
James Monroe

To extract just the first names, we can use the following command:

$ awk '{ print $1 }' presidents.txt

George
John
Thomas
James
James

Or, to just find those records with "James" as the first name:

$ awk '$1 ~ /James/ { print }' presidents.txt
James Madison
James Monroe

Awk can do a lot more, and features programming concepts such as 
variables, conditionals
and loops. But just a basic grasp of how to match and extract fields 
will get you far.

You might want to have a look at the /etc/init.d directory in Linux based systems. Click here.
Learn a new command today from this blog post.
To subscribe to the "Guy WhoSteals" feed, click here.
You can add yourself to the GuyWhoSteals fanpage on Facebook or follow GuyWhoSteals on Twitter.
Guy's personal blog: here.