Now days,
there are many ways to access a web page. We can directly access the web page
or we can download the web page using different tools.
In
this article we will see how we can access the web page using Perl which allows
downloading the web page data and processing it or changing it in the way we
need. There are many Perl libraries which actually do the processing of web
pages. In this article we will see a Perl module LWP::Simple
Basics
In
the below basic example,we are connecting to the google.com and downloading it
to a variable called content. Now when you print the variable we can see the
html elements of the web page.
#!/usr/bin/perl
use
strict;
use
warnings;
use
LWP::Simple;
my
$content = get('https://www.google.com') or die 'Unable to get page';
print
$content;
exit
0;
We can also
use the LWP::Simple to directly get the webpage or print the web page directly
to the STDOUT like
#!/usr/bin/perl
use
strict;
use
warnings;
use
LWP::Simple;
getprint('https://www.google.com’)
or die 'Unable to get page';
exit
0;
In another
way , we can make the web page downloaded it copied to a new file like,
#!/usr/bin/perl
use
strict;
use
warnings;
use
LWP::Simple;
getstore('https://www.google.com',
'test.html') or die 'Unable to get page';
exit
0;
This
downloaded the file into a test.html file.
Sometimes
it only makes sense to store a document
if it's been updated. We can do this with the mirror function, which
takes the same arguments as the getstore function:
mirror('http://google.com', 'google.html');
All
the above LWP Functions return the HTTP Status Code. We can use those to
compare like,
my
$response_code = getprint('https://google.com');print "nOKn" if
($response_code == RC_OK);
LWP in Object Oriented Way
If we
want to do more things with the web page, we can go with the LWP Object
oriented way using the LWP::UserAgent package.
#!/usr/bin/perl
use strict;
use warnings;
use LWP::UserAgent;
use HTTP::Request::Common qw(GET);
use HTTP::Cookies;
my $ua = LWP::UserAgent->new;
# Request object
my $req = GET 'https://www.google.com';
# Make the request
my $res = $ua->request($req);
#
Check the response
if ($res->is_success) {
print $res->status_line( );
} else {
print $res->status_line .
"\n";
}
exit 0;
The
above is the sample example which access the webpage ,but we only print the
Status of the Request.
The
first step is to define the Necessary packages like
use
LWP::UserAgent;
use
HTTP::Request::Common qw(GET);
use
HTTP::Cookies;
lets
define our User Agent and this is the object that acts as a browser and makes
requests and receives responses.
my
$ua = LWP::UserAgent->new;
Once
the agent is defined we can now define the request object which will be used to
request a url like
my
$req = GET 'https://www.google.com';
Since
we are using the HTTP::Request::Common module, we can use a GET method which
accepts a url as the first argument.
We can
also pass arguments like
my
$req = GET 'https://www.google.com' , [name => 'me', age => 24];
Once
the request object is defined, we can use the User Agent to make the request
like
my
$res = $ua->request($req);
The request method
returns a HTTP::Response object. This object contains the status code of the
response, and the content of the page if the request was successful.
We
can check the response of the Objects and we can also print the obtained
content
Basic
GET Example
#!/usr/bin/perl
use
LWP::UserAgent;
my
$ua = LWP::UserAgent->new;
my
$server_endpoint = "http://vx1379:10011/wam_wls_monitor";
# set
custom HTTP request header fields
my
$req = HTTP::Request->new(GET => $server_endpoint);
#$req->header('content-type'
=> 'application/json');
#$req->header('x-auth-token'
=> 'jklasdhjklsa');
my
$resp = $ua->request($req);
if
($resp->is_success) {
my $message = $resp->decoded_content;
print "Received reply:
$message\n";
}
else
{
print "HTTP GET error code: ",
$resp->code, "\n";
print "HTTP GET error message: ",
$resp->message, "\n";
}
Basic
POST Example
use
LWP::UserAgent;
my
$ua = LWP::UserAgent->new;
my
$server_endpoint = "http://vx1379:10011/wam_wls_monitor";
# set
custom HTTP request header fields
#my
$req = HTTP::Request->new(POST => $server_endpoint);
#$req->header('content-type'
=> 'application/json');
#$req->header('x-auth-token'
=> 'jklasdhjklsa');
my
$resp = $ua->request($req);
if
($resp->is_success) {
my $message = $resp->decoded_content;
print "Received reply:
$message\n";
}
else
{
print "HTTP GET error code: ",
$resp->code, "\n";
print "HTTP GET error message: ",
$resp->message, "\n";
}
More
To Come , Happy learning
