The Curl application is a tool to transfer data from a server using a number of different protocols. The main protocols that are generally used are HTTP and HTTPS.
There will be times when you need to download a web page, emulating a different browser. Some web sites may output different html code depending on the browser being used to request the page.
By modifying the User Agent header on the http/https request, you can tell the server what browser you are using.
Curl has the ability to modify the user agent value easily, using the “-a” parameter as shown in the example below.
Modifying the User Agent in Curl:
$ curl -A "User Agent Here" "http://website.url.here"
The example below will emulate Internet Explorer 6 requesting the page from www.google.com
$ curl -A "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" "http://www.google.com"
If the web page is designed to show different output depending on the browser, the above example would be show the output as though you were connecting to the web server from Internet Explorer 6.
For a list of some of the common user agent strings, please click here.
