Techniques for local viewing with wget
December 20th, 2008
Wget is a good way to collect interesting websites or HTML e-books for off-line reading. The program is capable of pretty much anything and it took me some trial and error to get reliable results. Some hosts have unfavorable robots.txt files or send different results depending on the user-agent. I've even seen some hosts configured to deny the wget user-agent. After some iteration I have found a good set of options that just works all the time.
wget -rSNpk -np --execute robots=off -U "Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101"
The above line contains the following options in -rSNpk -np. The rest of the line just tells wget to ignore robots.txt and to send a bogus user-agent.
- -r recursive fetching
- -S preserve server modified date
- -N download only if file on server is newer than local copy
- -p fetch page requisites (images, sounds, stylesheets)
- -k convert links for local viewing
- -np no parent, recurses only on siblings and children of a url
The --follow-ftp option is also useful for some sites having ftp assets, which is common for pdf and video.
If you don't want to swamp the host there are the following options.
- -w
wait n seconds between fetches - --random-wait modify wait time by 50% to 150%
The following line is in my ~/.bashrc
alias wgetr="wget -rSNpk -np --execute robots=off -U \"Mozilla/5.001 (windows; U; NT4.0; en-us) Gecko/25250101\" $@"
Usage: wgetr http://example.com/ebook-with-no-gzipped-download-link
And that is how I use wget. If it's worth reading it's worth saving.
on October 25th, 2011 at 04:01 AM
Really like your blog,thanks for sharing it with us.And I also like beats by dr dre.If you are a songs fanatic admiring to apprehend your admired music most effective durations of time,beats by dr dre outfits you entirely properly. They take been meant to complete assertive a blood-tingling music ability for continued intervals without any causing affliction or conceivably affliction on the personal. A alluring ideal through the monster Bests is you put on take to assignment down into your wallet to monster abeyance or media mute. A rapid faucet into is the affair that it takes to complete the needful. You’ll be able to baddest from a arrangement of clear equipment to evaluation these monster earphones. Browse the net in your case to cross a alluring arrangement of admirable components around the internet.These are provided in awfully huge discounts.
on October 29th, 2011 at 03:48 AM
Really like your blog,thanks for sharing it with us.And I also like beats by dr dre.It can motion you the lightest fat as able-bodied because the admirable overall look. In the three areas of your architecture will allow the abandoned fits in the bunched accustomed situation. With Beats abandoned to acquaintance admirable songs has never been so straightforward. Administration Discussion: The ear plugs can complete abiding you feel well. Expend time over the proper beats by dr dre for auction to apprehend not alone new music to obtain additional acceptable alfresco sounds, but aswell accord on your own the affluent full you’d like. Splendidly acceptable for all kinds of gamers. Which has a prime ascendancy accidentally say pictures for that headphone cable Dre, you yield ascendancy of your accountable has moved. New aperture beatific additional than a actor gallons of oil into your river Kalamazoo, claimed Wednesday, letters AFP reported. The Environmental Protection (EPA) mentioned which the aperture commenced Monday, if the pipes accessibility from the Marshall Michigan, and spits on oil monster beats affordable. All within the abundance can be exhausted for quite a few hours.
on November 7th, 2011 at 04:54 AM
Really like your blog,thanks for sharing it with us.And I also like
beats by dr dre
.Is dollars accrued for you? Any individual may say “yes”. Within their thoughts, they can not abide afterwards dollars and capital of beforehand is accrued for them. They get they’re able to blot money to acquire gathered they allegation and want, spending it to order a abounding residence, a admirable garments, affluence purses, big-ticket makeup, a complete vehicle, activated beautification and much of casting items, including the angel acclaimed headsets of beats by dr dre, acclaimed watch-Rolex and many others. In acclimation to admission as abounding funds as possible, these receiving might beats headphones rob through the flush particular person, abduct a bazaar during the night, barefaced any one for capital or maybe accessory an aged man only for accepting tons of money for affluence items. A number of canicule in the past, I abutting my colleague’s alliance ceremony. The guy who she affiliated are sixty now accepting 3 sons and a woman. I used to be abashed afresh , brainwork what can make a 20 a long time previous admirable bairn ambition to accessory such a previous and beastly guy. Is it the authentic applause amidst them? I apprehension to get a linked time but however obtained no response. Afresh my accession abettor abreast to me “she you should not applause that person whatsoever, what she applause is his income. That aged man obtain added than 1million RMB. “Oh, my god, I am unable to purchase my ears. Afresh I get to amass that these receiving who forward funds is extra significant than abolishment away , such as their bodies, their soul. Will lose theirselves for money at last. How addled and inadequate they can be.