3 WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize
5 =head1 How to get help with WWW::Mechanize
7 If your question isn't answered here in the FAQ, please turn to the
12 =item * L<http://perlmonks.org>
14 =item * The libwww-perl mailing list at L<http://lists.perl.org>
20 =head2 I have this web page that has JavaScript on it, and my Mech program doesn't work.
22 That's because WWW::Mechanize doesn't operate on the JavaScript. It only
23 understands the HTML parts of the page.
25 =head2 I thought Mech was supposed to work like a web browser.
27 It does pretty much, but it doesn't support JavaScript.
29 I added some basic attempts at picking up URLs in C<window.open()>
30 calls and return them in C<< $mech->links >>. They work sometimes.
32 Since Javascript is completely visible to the client, it cannot be used
33 to prevent a scraper from following links. But it can make life difficult. If
34 you want to scrape specific pages, then a solution is always possible.
36 One typical use of Javascript is to perform argument checking before
37 posting to the server. The URL you want is probably just buried in the
38 Javascript function. Do a regular expression match on
39 C<< $mech->content() >>
40 to find the link that you want and C<< $mech->get >> it directly (this
41 assumes that you know what you are looking for in advance).
43 In more difficult cases, the Javascript is used for URL mangling to
44 satisfy the needs of some middleware. In this case you need to figure
45 out what the Javascript is doing (why are these URLs always really
46 long?). There is probably some function with one or more arguments which
47 calculates the new URL. Step one: using your favorite browser, get the
48 before and after URLs and save them to files. Edit each file, converting
49 the the argument separators ('?', '&' or ';') into newlines. Now it is
50 easy to use diff or comm to find out what Javascript did to the URL.
51 Step 2 - find the function call which created the URL - you will need
52 to parse and interpret its argument list. The Javascript Debugger in the
53 Firebug extension for Firefox helps with the analysis. At this point, it is
54 fairly trivial to write your own function which emulates the Javascript
55 for the pages you want to process.
57 Here's annother approach that answers the question, "It works in Firefox,
58 but why not Mech?" Everything the web server knows about the client is
59 present in the HTTP request. If two requests are identical, the results
60 should be identical. So the real question is "What is different between
61 the mech request and the Firefox request?"
63 The Firefox extension "Tamper Data" is an effective tool for examining
64 the headers of the requests to the server. Compare that with what LWP
65 is sending. Once the two are identical, the action of the server should
68 I say "should", because this is an oversimplification - some values
69 are naturally unique, e.g. a SessionID, but if a SessionID is present,
70 that is probably sufficient, even though the value will be different
71 between the LWP request and the Firefox request. The server could use
72 the session to store information which is troublesome, but that's not
73 the first place to look (and highly unlikely to be relevant when you
74 are requesting the login page of your site).
76 Generally the problem is to be found in missing or incorrect POSTDATA
77 arguments, Cookies, User-Agents, Accepts, etc. If you are using mech,
78 then redirects and cookies should not be a problem, but are listed here
79 for completeness. If you are missing headers, C<< $mech->add_header >>
80 can be used to add the headers that you need.
82 =head2 Which modules work like Mechanize and have JavaScript support?
84 In no particular order: L<Gtk2::WebKit::Mechanize>, L<Win32::IE::Mechanize>,
85 L<WWW::Mechanize::Firefox>, L<WWW::Scripter>, L<WWW::Selenium>
89 =head2 Can I do [such-and-such] with WWW::Mechanize?
91 If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is
92 a subclass of L<LWP::UserAgent>, so all the wondrous magic of that
95 =head2 How do I use WWW::Mechanize through a proxy server?
97 See the docs in L<LWP::UserAgent> on how to use the proxy. Short version:
99 $mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');
101 or get the specs from the environment:
105 # Environment set like so:
106 gopher_proxy=http://proxy.my.place/
107 wais_proxy=http://proxy.my.place/
108 no_proxy="localhost,my.domain"
109 export gopher_proxy wais_proxy no_proxy
111 =head2 How can I see what fields are on the forms?
113 Use the mech-dump utility, optionally installed with Mechanize.
115 $ mech-dump --forms http://search.cpan.org
117 GET http://search.cpan.org/search
119 mode=all (option) [*all|module|dist|author]
120 <NONAME>=CPAN Search (submit)
122 =head2 How do I get Mech to handle authentication?
126 my $agent = WWW::Mechanize->new();
128 Authorization => "Basic " .
129 MIME::Base64::encode( USER . ':' . PASS )
132 $agent->credentials( ADDRESS, REALM, USER, PASS );
133 $agent->get( URL, @args );
135 If you want to use the credentials for all future requests, you can
136 also use the L<LWP::UserAgent> C<default_header()> method instead
137 of the extra arguments to C<get()>
139 $mech->default_header(
140 Authorization => 'Basic ' . encode_base64( USER . ':' . PASSWORD ) );
142 =head2 How can I get WWW::Mechanize to execute this JavaScript?
144 You can't. JavaScript is entirely client-based, and WWW::Mechanize
145 is a client that doesn't understand JavaScript. See the top part
148 =head2 How do I check a checkbox that doesn't have a value defined?
150 Set it to to the value of "on".
152 $mech->field( my_checkbox => 'on' );
154 =head2 How do I handle frames?
156 You don't deal with them as frames, per se, but as links. Extract
159 my @frame_links = $mech->find_link( tag => "frame" );
161 =head2 How do I get a list of HTTP headers and their values?
163 All L<HTTP::Headers> methods work on a L<HTTP::Response> object which is
164 returned by the I<get()>, I<reload()>, I<response()/res()>, I<click()>,
165 I<submit_form()>, and I<request()> methods.
167 my $mech = WWW::Mechanize->new( autocheck => 1 );
168 $mech->get( 'http://my.site.com' );
169 my $res = $mech->response();
170 for my $key ( $response->header_field_names() ) {
171 print $key, " : ", $response->header( $key ), "\n";
174 =head2 How do I enable keep-alive?
176 Since L<WWW::Mechanize> is a subclass of L<LWP::UserAgent>, you can
177 use the same mechanism to enable keep-alive:
181 $mech->conn_cache(LWP::ConnCache->new);
183 =head2 How can I change/specify the action parameter of an HTML form?
185 You can access the action of the form by utilizing the L<HTML::Form>
186 object returned from one of the specifying form methods.
188 Using C<< $mech->form_number($number) >>:
190 my $mech = WWW::mechanize->new;
191 $mech->get('http://someurlhere.com');
192 # Access the form using its Zero-Based Index by DOM order
193 $mech->form_number(0)->action('http://newAction'); #ABS URL
195 Using C<< $mech->form_name($number) >>:
197 my $mech = WWW::mechanize->new;
198 $mech->get('http://someurlhere.com');
199 #Access the form using its Zero-Based Index by DOM order
200 $mech->form_name('trgForm')->action('http://newAction'); #ABS URL
202 =head2 How do I save an image? How do I save a large tarball?
204 An image is just content. You get the image and save it.
206 $mech->get( 'photo.jpg' );
207 $mech->save_content( '/path/to/my/directory/photo.jpg' );
209 You can also save any content directly to disk using the C<:content_file>
210 flag to C<get()>, which is part of L<LWP::UserAgent>.
212 $mech->get( 'http://www.cpan.org/src/stable.tar.gz',
213 ':content_file' => 'stable.tar.gz' );
215 =head2 How do I pick a specific value from a C<< <select> >> list?
217 Find the C<HTML::Form::ListInput> in the page.
219 my ($listbox) = $mech->find_all_inputs( name => 'listbox' );
221 Then create a hash for the lookup:
224 @name_lookup{ $listbox->value_names } = $listbox->possible_values;
225 my $value = $name_lookup{ 'Name I want' };
227 If you have duplicate names, this method won't work, and you'll
228 have to loop over C<< $listbox->value_names >> and
229 C<< $listbox->possible_values >> in parallel until you find a
232 =head2 How do I get Mech to not follow redirects?
234 You use functionality in LWP::UserAgent, not Mech itself.
236 $mech->requests_redirectable( [] );
238 Or you can set C<max_redirect>:
240 $mech->max_redirect( 0 );
242 Both these options can also be set in the constructor. Mech doesn't
243 understand them, so will pass them through to the LWP::UserAgent
247 =head1 Why doesn't this work: Debugging your Mechanize program
249 =head2 My Mech program doesn't work, but it works in the browser.
251 Mechanize acts like a browser, but apparently something you're doing
252 is not matching the browser's behavior. Maybe it's expecting a
253 certain web client, or maybe you've not handling a field properly.
254 For some reason, your Mech problem isn't doing exactly what the
255 browser is doing, and when you find that, you'll have the answer.
257 =head2 My Mech program gets these 500 errors.
259 A 500 error from the web server says that the program on the server
260 side died. Probably the web server program was expecting certain
261 inputs that you didn't supply, and instead of handling it nicely,
264 Whatever the cause of the 500 error, if it works in the browser,
265 but not in your Mech program, you're not acting like the browser.
266 See the previous question.
268 =head2 Why doesn't my program handle this form correctly?
270 Run F<mech-dump> on your page and see what it says.
272 F<mech-dump> is a marvelous diagnostic tool for figuring out what forms
273 and fields are on the page. Say you're scraping CNN.com, you'd get this:
275 $ mech-dump http://www.cnn.com/
276 GET http://search.cnn.com/cnn/search
277 source=cnn (hidden readonly)
278 invocationType=search/top (hidden readonly)
279 sites=web (radio) [*web/The Web ??|cnn/CNN.com ??]
281 <NONAME>=Search (submit)
283 POST http://cgi.money.cnn.com/servlets/quote_redirect
285 <NONAME>=GET (submit)
287 POST http://polls.cnn.com/poll
288 poll_id=2112 (hidden readonly)
289 question_1=<UNDEF> (radio) [1/Simplistic option|2/VIEW RESULTS]
290 <NONAME>=VOTE (submit)
292 GET http://search.cnn.com/cnn/search
293 source=cnn (hidden readonly)
294 invocationType=search/bottom (hidden readonly)
295 sites=web (radio) [*web/??CNN.com|cnn/??]
297 <NONAME>=Search (submit)
299 Four forms, including the first one duplicated at the end. All the
300 fields, all their defaults, lovingly generated by HTML::Form's C<dump>
303 If you want to run F<mech-dump> on something that doesn't lend itself
304 to a quick URL fetch, then use the C<save_content()> method to write
305 the HTML to a file, and run F<mech-dump> on the file.
307 =head2 Why don't https:// URLs work?
309 You need either L<IO::Socket::SSL> or L<Crypt::SSLeay> installed.
311 =head2 Why do I get "Input 'fieldname' is readonly"?
313 You're trying to change the value of a hidden field and you have
316 First, make sure that you actually mean to change the field that you're
317 changing, and that you don't have a typo. Usually, hidden variables are
318 set by the site you're working on for a reason. If you change the value,
319 you might be breaking some functionality by faking it out.
321 If you really do want to change a hidden value, make the changes in a
322 scope that has warnings turned off:
326 $agent->field( name => $value );
329 =head2 I tried to [such-and-such] and I got this weird error.
331 Are you checking your errors?
335 Are you checking that your action succeeded after every action?
339 For example, if you try this:
341 $mech->get( "http://my.site.com" );
342 $mech->follow_link( "foo" );
344 and the C<get> call fails for some reason, then the Mech internals
345 will be unusable for the C<follow_link> and you'll get a weird
346 error. You B<must>, after every action that GETs or POSTs a page,
347 check that Mech succeeded, or all bets are off.
349 $mech->get( "http://my.site.com" );
350 die "Can't even get the home page: ", $mech->response->status_line
351 unless $mech->success;
353 $mech->follow_link( "foo" );
354 die "Foo link failed: ", $mech->response->status_line
355 unless $mech->success;
357 =head2 How do I figure out why C<< $mech->get($url) >> doesn't work?
359 There are many reasons why a C<< get() >> can fail. The server can take
360 you to someplace you didn't expect. It can generate redirects which are
361 not properly handled. You can get time-outs. Servers are down more often
362 than you think! etc, etc, etc. A couple of places to start:
366 =item 1 Check C<< $mech->status() >> after each call
368 =item 2 Check the URL with C<< $mech->uri() >> to see where you ended up
370 =item 3 Try debugging with C<< LWP::Debug >>.
374 If things are really strange, turn on debugging with
375 C<< use LWP::Debug qw(+); >>
376 Just put this in the main program. This causes LWP to print out a trace
377 of the HTTP traffic between client and server and can be used to figure
378 out what is happening at the protocol level.
380 It is also useful to set many traps to verify that processing is
381 proceeding as expected. A Mech program should always have an "I didn't
382 expect to get here" or "I don't recognize the page that I am processing"
385 Since errors can be transient, by the time you notice that the error
386 has occurred, it might not be possible to reproduce it manually. So
387 for automated processing it is useful to email yourself the following
392 =item * where processing is taking place
394 =item * An Error Message
398 =item * $mech->content
402 You can also save the content of the page with C<< $mech->save_content( 'filename.html' ); >>
404 =head2 I submitted a form, but the server ignored everything! I got an empty form back!
406 The post is handled by application software. It is common for PHP
407 programmers to use the same file both to display a form and to process
408 the arguments returned. So the first task of the application programmer
409 is to decide whether there are arguments to processes. The program can
410 check whether a particular parameter has been set, whether a hidden
411 parameter has been set, or whether the submit button has been clicked.
412 (There are probably other ways that I haven't thought of).
414 In any case, if your form is not setting the parameter (e.g. the submit
415 button) which the web application is keying on (and as an outsider there
416 is no way to know what it is keying on), it will not notice that the form
417 has been submitted. Try using C<< $mech->click() >> instead of
418 C<< $mech->submit() >> or vice-versa.
420 =head2 I've logged in to the server, but I get 500 errors when I try to get to protected content.
422 Some web sites use distributed databases for their processing. It
423 can take a few seconds for the login/session information to percolate
424 through to all the servers. For human users with their slow reaction
425 times, this is not a problem, but a Perl script can outrun the server.
426 So try adding a C<sleep(5)> between logging in and actually doing anything
427 (the optimal delay must be determined experimentally).
429 =head2 Mech is a big memory pig! I'm running out of RAM!
431 Mech keeps a history of every page, and the state it was in. It actually
432 keeps a clone of the full Mech object at every step along the way.
434 You can limit this stack size with the C<stack_depth> parm in the C<new()>
435 constructor. If you set stack_size to 0, Mech will not keep any history.
439 Copyright 2005-2009 Andy Lester C<< <andy at petdance.com> >>