The Wget application is a non-interactive tool to download data from a server using either the HTTP, HTTPS or FTP protocols. This article will focus on the HTTP and HTTPS protocols, as the user agent value has no relevance with the FTP protocol.
There will be times when you need to download a web page, emulating a different browser. Some web sites may output different html code depending on the browser being used to request the page.
By modifying the User Agent header on the http/https request, you can tell the server what browser you are using.
Wget has the ability to modify the user agent value easily, using the “-a” parameter as shown in the example below.
Modifying the User Agent in Wget:
$ wget -U "User Agent Here" "http://website.url.here"
The example below will emulate Internet Explorer 6 requesting the page from www.google.com
$ wget -U "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"
or
$ wget --user-agent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"
If the web page is designed to show different output depending on the browser, the above example would be show the output as though you were connecting to the web server from Internet Explorer 6.
For a list of some of the common user agent strings, please click here.
