Class HTTPClient
In: lib/httpclient/connection.rb
lib/httpclient/auth.rb
lib/httpclient/util.rb
lib/httpclient/session.rb
lib/httpclient/ssl_config.rb
lib/httpclient/timeout.rb
lib/httpclient.rb
Parent: Object

The HTTPClient class provides several methods for accessing Web resources via HTTP.

HTTPClient instance is designed to be MT-safe. You can call a HTTPClient instance from several threads without synchronization after setting up an instance.

  clnt = HTTPClient.new
  clnt.set_cookie_store('/home/nahi/cookie.dat')
  urls.each do |url|
    Thread.new(url) do |u|
      p clnt.head(u).status
    end
  end

How to use

At first, how to create your client. See initialize for more detail.

  1. Create simple client.
     clnt = HTTPClient.new
    
  2. Accessing resources through HTTP proxy. You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ instead.
     clnt = HTTPClient.new('http://myproxy:8080')
    

How to retrieve web resources

See get_content.

  1. Get content of specified URL. It returns a String of whole result.
     puts clnt.get_content('http://dev.ctor.org/')
    
  2. Get content as chunks of String. It yields chunks of String.
     clnt.get_content('http://dev.ctor.org/') do |chunk|
       puts chunk
     end
    

Invoking other HTTP methods

See head, get, post, put, delete, options, propfind, proppatch and trace. It returns a HTTP::Message instance as a response.

  1. Do HEAD request.
     res = clnt.head(uri)
     p res.header['Last-Modified'][0]
    
  2. Do GET request with query.
     query = { 'keyword' => 'ruby', 'lang' => 'en' }
     res = clnt.get(uri, query)
     p res.status
     p res.contenttype
     p res.header['X-Custom']
     puts res.content
    

How to POST

See post.

  1. Do POST a form data.
     body = { 'keyword' => 'ruby', 'lang' => 'en' }
     res = clnt.post(uri, body)
    
  2. Do multipart file upload with POST. No need to set extra header by yourself from httpclient/2.1.4.
     File.open('/tmp/post_data') do |file|
       body = { 'upload' => file, 'user' => 'nahi' }
       res = clnt.post(uri, body)
     end
    

Accessing via SSL

Ruby needs to be compiled with OpenSSL.

  1. Get content of specified URL via SSL. Just pass an URL which starts with ‘https://’.
     https_url = 'https://www.rsa.com'
     clnt.get_content(https_url)
    
  2. Getting peer certificate from response.
     res = clnt.get(https_url)
     p res.peer_cert #=> returns OpenSSL::X509::Certificate
    
  3. Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.
     user_cert_file = 'cert.pem'
     user_key_file = 'privkey.pem'
     clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
     clnt.get_content(https_url)
    

Handling Cookies

  1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.
     clnt = HTTPClient.new
     clnt.get_content(url1) # receives Cookies.
     clnt.get_content(url2) # sends Cookies if needed.
    
  2. Saving non volatile Cookies to a specified file. Need to set a file at first and invoke save method at last.
     clnt = HTTPClient.new
     clnt.set_cookie_store('/home/nahi/cookie.dat')
     clnt.get_content(url)
     ...
     clnt.save_cookie_store
    
  3. Disabling Cookies.
     clnt = HTTPClient.new
     clnt.cookie_manager = nil
    

Configuring authentication credentials

  1. Authentication with Web server. Supports BasicAuth, DigestAuth, and Negotiate/NTLM (requires ruby/ntlm module).
     clnt = HTTPClient.new
     domain = 'http://dev.ctor.org/http-access2/'
     user = 'user'
     password = 'user'
     clnt.set_auth(domain, user, password)
     p clnt.get_content('http://dev.ctor.org/http-access2/login').status
    
  2. Authentication with Proxy server. Supports BasicAuth and NTLM (requires win32/sspi)
     clnt = HTTPClient.new(proxy)
     user = 'proxy'
     password = 'proxy'
     clnt.set_proxy_auth(user, password)
     p clnt.get_content(url)
    

Invoking HTTP methods with custom header

Pass a Hash or an Array for extheader argument.

    extheader = { 'Accept' => '*/*' }
    clnt.get_content(uri, query, extheader)

    extheader = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
    clnt.get_content(uri, query, extheader)

Invoking HTTP methods asynchronously

See head_async, get_async, post_async, put_async, delete_async, options_async, propfind_async, proppatch_async, and trace_async. It immediately returns a HTTPClient::Connection instance as a returning value.

    connection = clnt.post_async(url, body)
    print 'posting.'
    while true
      break if connection.finished?
      print '.'
      sleep 1
    end
    puts '.'
    res = connection.pop
    p res.status
    p res.content.read # res.content is an IO for the res of async method.

Shortcut methods

You can invoke get_content, get, etc. without creating HTTPClient instance.

  ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
  ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/

Methods

Included Modules

Util

Classes and Modules

Module HTTPClient::DebugSocket
Module HTTPClient::SocketWrap
Module HTTPClient::Timeout
Module HTTPClient::Util
Class HTTPClient::AuthFilterBase
Class HTTPClient::BadResponseError
Class HTTPClient::BasicAuth
Class HTTPClient::ConfigurationError
Class HTTPClient::ConnectTimeoutError
Class HTTPClient::Connection
Class HTTPClient::DigestAuth
Class HTTPClient::LoopBackSocket
Class HTTPClient::NegotiateAuth
Class HTTPClient::ProxyAuth
Class HTTPClient::ReceiveTimeoutError
Class HTTPClient::SSLConfig
Class HTTPClient::SSLSocketWrap
Class HTTPClient::SSPINegotiateAuth
Class HTTPClient::SendTimeoutError
Class HTTPClient::Session
Class HTTPClient::SessionManager
Class HTTPClient::Site
Class HTTPClient::TimeoutError
Class HTTPClient::TimeoutScheduler
Class HTTPClient::WWWAuth

Constants

NTLMEnabled = true
NTLMEnabled = false
SSPIEnabled = true
SSPIEnabled = false
SSLEnabled = true
SSLEnabled = false
VERSION = '2.1.5'
RUBY_VERSION_STRING = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
LIB_NAME = "(#{$1}/#{$2}, #{RUBY_VERSION_STRING})"
PROPFIND_DEFAULT_EXTHEADER = { 'Depth' => '0' }   Default extheader for PROPFIND request.
NO_PROXY_HOSTS = ['localhost']

Attributes

cookie_manager  [RW] 
WebAgent::CookieManager:Cookies configurator.
follow_redirect_count  [RW]  How many times get_content and post_content follows HTTP redirect. 10 by default.
proxy_auth  [R] 
HTTPClient::ProxyAuth:Proxy authentication handler.
request_filter  [R]  An array of request filter which can trap HTTP request/response. See HTTPClient::WWWAuth to see how to use it.
ssl_config  [R] 
HTTPClient::SSLConfig:SSL configurator.
test_loopback_response  [R]  An array of response HTTP message body String which is used for loop-back test. See test/* to see how to use it. If you want to do loop-back test of HTTP header, use test_loopback_http_response instead.
www_auth  [R] 
HTTPClient::WWWAuth:WWW authentication handler.

Public Class methods

Creates a HTTPClient instance which manages sessions, cookies, etc.

HTTPClient.new takes 3 optional arguments for proxy url string, User-Agent String and From header String. User-Agent and From are embedded in HTTP request Header if given. No User-Agent and From header added without setting it explicitly.

  proxy = 'http://myproxy:8080'
  agent_name = 'MyAgent/0.1'
  from = 'from@example.com'
  HTTPClient.new(proxy, agent_name, from)

You can use a keyword argument style Hash. Keys are :proxy, :agent_name and :from.

  HTTPClient.new(:agent_name = 'MyAgent/0.1')

CAUTION: caller must aware of race condition.

Public Instance methods

Returns debug device if exists. See debug_dev=.

Sets debug device. Once debug device is set, all HTTP requests and responses are dumped to given device. dev must respond to << for dump.

Calling this method resets all existing sessions.

A default method for redirect uri callback. This method is used by HTTPClient instance by default. This callback allows relative redirect such as

  Location: ../foo/

in HTTP header.

Sends DELETE request to the specified URL. See request for arguments.

Sends DELETE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Sends GET request to the specified URL. See request for arguments.

Sends GET request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Retrieves a web resource.

uri:a String or an URI object which represents an URL of web resource.
query:a Hash or an Array of query part of URL. e.g. { "a" => "b" } => ‘host/part?a=b’. Give an array to pass multiple value like
["a", "b"], ["a", "c"]
=> ‘host/part?a=b&a=c’.
extheader:a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block:Give a block to get chunked message-body of response like get_content(uri) { |chunked_body| … }. Size of each chunk may not be the same.

get_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to retrieve content from redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use get method. get returns HTTP::Message as a response and you need to follow HTTP redirect by yourself if you need.

Sends HEAD request to the specified URL. See request for arguments.

Sends HEAD request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Returns NO_PROXY setting String if given.

Sets NO_PROXY setting String. no_proxy must be a comma separated String. Each entry must be ‘host’ or ‘host:port’ such as; HTTPClient#no_proxy = ‘example.com,example.co.jp:443‘

‘localhost’ is treated as a no_proxy site regardless of explicitly listed. HTTPClient checks given URI objects before accessing it. ‘host’ is tail string match. No IP-addr conversion.

You can use environment variable ‘no_proxy’ or ‘NO_PROXY’ for it.

Calling this method resets all existing sessions.

Sends OPTIONS request to the specified URL. See request for arguments.

Sends OPTIONS request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Sends POST request to the specified URL. See request for arguments.

Sends POST request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Posts a content.

uri:a String or an URI object which represents an URL of web resource.
body:a Hash or an Array of body part. e.g. { "a" => "b" } => ‘a=b’. Give an array to pass multiple value like
["a", "b"], ["a", "c"]
=> ‘a=b&a=c’.

When you pass a File as a value, it will be posted as a multipart/form-data. e.g. { ‘upload’ => file }

extheader:a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block:Give a block to get chunked message-body of response like post_content(uri) { |chunked_body| … }. Size of each chunk may not be the same.

post_content follows HTTP redirect status (see HTTP::Status.redirect?) internally and try to post the content to redirected URL. See redirect_uri_callback= how HTTP redirection is handled.

If you need to get full HTTP response including HTTP status and headers, use post method.

Sends PROPFIND request to the specified URL. See request for arguments.

Sends PROPFIND request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Sends PROPPATCH request to the specified URL. See request for arguments.

Sends PROPPATCH request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Returns URI object of HTTP proxy if exists.

Sets HTTP proxy used for HTTP connection. Given proxy can be an URI, a String or nil. You can set user/password for proxy authentication like HTTPClient#proxy = ‘user:passwd@myproxy:8080

You can use environment variable ‘http_proxy’ or ‘HTTP_PROXY’ for it. You need to use ‘cgi_http_proxy’ or ‘CGI_HTTP_PROXY’ instead if you run HTTPClient from CGI environment from security reason. (HTTPClient checks ‘REQUEST_METHOD’ environment variable whether it‘s CGI or not)

Calling this method resets all existing sessions.

Sends PUT request to the specified URL. See request for arguments.

Sends PUT request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

Sets callback proc when HTTP redirect status is returned for get_content and post_content. default_redirect_uri_callback is used by default.

If you need strict implementation which does not allow relative URI redirection, set strict_redirect_uri_callback instead.

  clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

Sends a request to the specified URL.

method:HTTP method to be sent. method.to_s.upcase is used.
uri:a String or an URI object which represents an URL of web resource.
query:a Hash or an Array of query part of URL. e.g. { "a" => "b" } => ‘host/part?a=b’ Give an array to pass multiple value like
["a", "b"], ["a", "c"]
=> ‘host/part?a=b&a=c
body:a Hash or an Array of body part. e.g. { "a" => "b" } => ‘a=b’. Give an array to pass multiple value like
["a", "b"], ["a", "c"]
=> ‘a=b&a=c’.

When the given method is ‘POST’ and the given body contains a file as a value, it will be posted as a multipart/form-data. e.g. { ‘upload’ => file } See HTTP::Message.file? for actual condition of ‘a file’.

extheader:a Hash or an Array of extra headers. e.g. { ‘Accept’ => ’*/*’ } or [[‘Accept’, ‘image/jpeg’], [‘Accept’, ‘image/png’]].
&block:Give a block to get chunked message-body of response like get(uri) { |chunked_body| … }. Size of each chunk may not be the same.

You can also pass a String as a body. HTTPClient just sends a String as a HTTP request message body.

When you pass an IO as a body, HTTPClient sends it as a HTTP request with chunked encoding (Transfer-Encoding: chunked in HTTP header). Bear in mind that some server application does not support chunked request. At least cgi.rb does not support it.

Sends a request in async style. request method creates new Thread for HTTP connection and returns a HTTPClient::Connection instance immediately.

Arguments definition is the same as request.

Resets internal session for the given URL. Keep-alive connection for the site (host-port pair) is disconnected if exists.

Resets all of internal sessions. Keep-alive connections are disconnected.

Try to save Cookies to the file specified in set_cookie_store. Unexpected error will be raised if you don‘t call set_cookie_store first. (interface mismatch between WebAgent::CookieManager implementation)

Sets credential for Web server authentication.

domain:a String or an URI to specify where HTTPClient should use this
      credential.  If you set uri to nil, HTTPClient uses this credential
      wherever a server requires it.
user:username String.
passwd:password String.

You can set multiple credentials for each uri.

  clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
  clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')

Calling this method resets all existing sessions.

Sets the filename where non-volatile Cookies be saved by calling save_cookie_store. This method tries to load and managing Cookies from the specified file.

Calling this method resets all existing sessions.

Sets credential for Proxy authentication.

user:username String.
passwd:password String.

Calling this method resets all existing sessions.

A method for redirect uri callback. How to use:

  clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)

This callback does not allow relative redirect such as

  Location: ../foo/

in HTTP header. (raises BadResponseError instead)

Sends TRACE request to the specified URL. See request for arguments.

Sends TRACE request in async style. See request_async for arguments. It immediately returns a HTTPClient::Connection instance as a result.

[Validate]