|
|
|
  |
Good news - I found more stable alternatives to my programs. Jump to the very bottom of this page to see what I found. Or you're still welcome to read on and get the tools I wrote...I needed to automate the download of data from a web site. There didn't seem to be any way to do it. So I had to write a program. WebGet can be run from a DOS batch file or from the Windows Scripting Host (Or VBA or any ActiveX client). It can retrieve text or binary files under both the HTTP and FTP protocols. Although it is technically a Windows program, there is no user interface. Click the "Download Now!" icon to get the WebGet program, source code, and sample DOS and VBScript code.
I hate to mention this, but some users have reported (and I've confirmed) that there are some webservers that the WebGet program seems to retrieve only the first 1 or 2 kb of data from. This is due to "chunking", caused by server programs that send data without buffering it. This causes the socket to keep closing and opening. I only get the first chunk. But basically it works for me on the servers I use. So there.
Here's my compromise. I had another project I was working on. It retrieves http files (no ftp), but it doesn't use the MSINET control that is susceptible to the chunking problem. It uses the MSWINSCK control, and even though it is still very basic, it is way more stable. I call it "Post" (I don't bother trying to come up with creative names), and you can get it here. It works with both the http "POST" and "GET" methods. The "WebGet" program only did "GET".
Like everything I have, it comes with samples and source code. And there's a good chance it will solve your problem if you need to automate the download of web pages.
The above controls were written for Windows 95 and 98. Luckily,
Windows 2000 and newer (if you've kept up with the service packs) now
have a built-in scriptable object for web downloading. Try code like
this for a start:
Dim web, strUrl, strWebPage
Set web = CreateObject("WinHttp.WinHttpRequest.5.1")
web.Open "GET", strURL, False
web.Send
strWebPage = web.ResponseText
The WinHttp object can do everything. It handles binary downloads,
authentication, and redirects. I've found a few other methods to allow
me to download things with Windows Scripting. I provide links to them
here:
http://www.ericphelps.com/scripting/index.htm#Download
If you're working in batch, the command-line programs WGET, Pavuk, and cURL can download virtually anything using any method. They were all written primarily for Unix, but all have compiled Windows binaries if you look around.
WGET (My favorite version is 1.8.1):
http://wget.sunsite.dk/
http://space.tin.it/computer/hherold/
http://www.interlog.com/~tcharron/wgetwin.html
ftp://ftp.newcastle.edu.au/pub/nt/gnu/wget.zip
Pavuk:
http://pavuk.sourceforge.net/
cURL:
http://curl.haxx.se/
Lost? Look at the site map.
Bad links? Questions? Send me mail.