Abstract
In this chapter, we focus on general, all-purpose infrastructure we can use in R for accessing networks and the Web. The RCurl package provides both high-level and intermediate-level functionality within R that allow us to make rich and flexible requests to a large variety of different servers, including Web servers and applications that speak different protocols. RCurl generalizes the functionality built into R for downloading documents, etc., by a) supporting more protocols, b) allowing us to control many more aspects of the requests.We use RCurl as the foundation in R for scraping data from static Web pages, submitting HTML forms, interacting with RESTful APIs (Application Programming Interfaces) on both Web servers and local applications, invoking methods provided by SOAP servers, and using XML-RPC for remote procedure calls. RCurl provides support for many protocols used on the Internet (e.g., HTTP, HTTPS, FTP) and a large collection of options for controlling the requests (e.g., cookies, login and passwords, content type, header fields). In this chapter, we will focus primarily on HTTP and illustrate how to use RCurl generally by developing interfaces to different Web APIs.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Daniel Adler. rdyncall: Improved foreign function interface (FFI) and dynamic bindings to C libraries (e.g., OpenGL). http://cran.r-project.org/package=rdyncall, 2012. R package version 0.7.5.
R. Fielding, J. Gettys, J. Mugul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee. Hypertext Transfer Protocol: HTTP/1.1. Worldwide Web Consortium, 1999. http://www.w3.org/Protocols/rfc2616/rfc2616.html.
A. Frank and A. Asuncion. UCI machine learning repository. http://archive.ics.uci.edu/ml, 2010.
Robert Gentleman and Ross Ihaka. Lexical scope and statistical computing. Journal of Computational and Graphical Statistics, 9:491–508, 2000.
GlobalGiving Foundation. Globalgiving: Donate to projects around the world supporting disaster relief, education, health, women and children, and more. http://www.globalgiving.org/, 2012.
David Gourley and Brian Totty. HTTP: The Definitive Guide. O’Reilly Media, Inc., Sebastopol, CA, 2002.
Michael Nielsen. Reinventing Discovery: The New Era of Networked Science. Princeton University Press, Princeton, NJ, 2012.
Open SSL Project. Open SSL: Cryptography and SSL/TLS toolkit. http://openssl.org,2011.
Karthik Ram and Duncan Temple Lang. rDrop: Dropbox R interface. https://github.com/karthikram/rDrop/, 2012. R package version 0.3.
Leonard Richardson and Sam Ruby. RESTful Web Services. O’Reilly Media, Inc., Sebastopol, CA, 2007.
Daniel Steinberg. curl easy setopt — Set options for a curl easy handle. http://curl.haxx.se/libcurl/c/curl_easy_setopt.html, 2012.
Daniel Steinberg. libcurl: The multiprotocol file transfer library. http://curl.haxx.se, 2012.
Duncan Temple Lang. RGCCTranslationUnit: R interface to GCC source code information. http://www.omegahat.org/RGCCTranslationUnit, 2009. R package version 0.4-
Duncan Temple Lang. RGoogleTrends: Download Google Trends data. http://www.omegahat.org/RGoogleTrends, 2009. R package version 0.2-1.
Duncan Temple Lang. RCIndex: R interface to the clang parser’s C API. http://www.omegahat.org/RCIndex, 2010. R package version 0.2-0.
Duncan Temple Lang. Rffi: Interface to libffi to dynamically invoke arbitrary compiled routines at run-time without compiled bindings. http://www.omegahat.org/Rffi, 2011. R package version 0.3-0.
Duncan Temple Lang. XML: Tools for parsing and generating XML within R and S-PLUS. http://www.omegahat.org/RSXML, 2011. R package version 3.4.
Duncan Temple Lang. Rcompression: In-memory decompression for GNU zip and bzip2 formats. http://www.omegahat.org/Rcompression, 2012. R package version 0.94-0.
Duncan Temple Lang. RCurl: General network (HTTP, FTP, etc.) client interface for R. http://www.omegahat.org/RCurl, 2012. R package version 1.95-3.
Duncan Temple Lang. RGoogleDocs: Primitive interface to Google Documents from R. http://www.omegahat.org/RGoogleDocs, 2012. R package version 0.7-0.
Duncan Temple Lang. SSOAP: Client-side SOAP access for R. http://www.omegahat.org/SSOAP, 2012. R package version 0.9-0.
Twitter, Inc. Twitter: A real-time information network. http://twitter.com/about, 2012.
Dave Winer. XML-RPC specification. http://xmlrpc.scripting.com/spec.html, 1999.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this chapter
Cite this chapter
Nolan, D., Lang, D.T. (2014). HTTP Requests. In: XML and Web Technologies for Data Sciences with R. Use R!. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7900-0_8
Download citation
DOI: https://doi.org/10.1007/978-1-4614-7900-0_8
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-7899-7
Online ISBN: 978-1-4614-7900-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)