node-modulesurllib
Request HTTP URLs in a complex world — basic and digest authentication, redirections, timeout and more.
Install
npm install urllib
Usage
TypeScript and ESM
import { request } from 'urllib';
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
CommonJS
const { request } = require('urllib');
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
API Doc
Method: async request(url[, options])
Arguments
- url String | Object - The URL to request, either a String or a Object that return by url.parse.
- options Object - Optional
- method String - Request method, defaults to
GET. Could beGET,POST,DELETEorPUT. Alias 'type'. - data Object - Data to be sent. Will be stringify automatically.
- content String | Buffer - Manually set the content of payload. If set,
datawill be ignored. - stream stream.Readable - Stream to be pipe to the remote. If set,
dataandcontentwill be ignored. - writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and
callbackwill be called withdatasetnullafter finished writing. - files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with
multipart/form-dataformat, base onformstream. Ifmethodnot set, will usePOSTmethod by default. - contentType String - Type of request data. Could be
json(Notes: not useapplication/jsonhere). If it'sjson, will auto setContent-Type: application/jsonheader. - dataType String - Type of response data. Could be
textorjson. If it'stext, thecallbackeddatawould be a String. If it'sjson, thedataof callback would be a parsed JSON Object and will auto setAccept: application/jsonheader. Defaultcallbackeddatawould be aBuffer. - fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is
false. - headers Object - Request headers.
- timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Default is
5000. You can usetimeout: 5000to tell urllib use same timeout on two phase or set them seperately such astimeout: [3000, 5000], which will set connecting timeout to 3s and response 5s. - keepAliveTimeout
number | null- Default is4000, 4 seconds - The timeout after which a socket without active requests will time out. Monitors time between activity on a connected socket. This value may be overridden by keep-alive hints from the server. See MDN: HTTP - Headers - Keep-Alive directives for more details. - auth String -
username:passwordused in HTTP Basic Authorization. - digestAuth String -
username:passwordused in HTTP Digest Authorization. - followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to true.
- maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
- formatRedirectUrl Function - Format the redirect url by your self. Default is
url.resolve(from, to). - beforeRequest Function - Before request hook, you can change every thing here.
- streaming Boolean - let you get the
resobject when request connected, defaultfalse. aliascustomResponse - compressed Boolean - Accept
gzip, brresponse content and auto decode it, default istrue. - timing Boolean - Enable timing or not, default is
true. - socketPath String | null - request a unix socket service, default is
null. - highWaterMark Number - default is
67108864, 64 KiB.
- method String - Request method, defaults to
Options: options.data
When making a request:
await request('https://example.com', {
method: 'GET',
data: {
'a': 'hello',
'b': 'world',
},
});
For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.
For others like POST, PATCH or PUT request,
in defaults, the data will be stringify into application/x-www-form-urlencoded format
if content-type header is not set.
If content-type is application/json, the data will be JSON.stringify to JSON data format.
Options: options.content
options.content is useful when you wish to construct the request body by yourself,
for example making a content-type: application/json request.
Notes that if you want to send a JSON body, you should stringify it yourself:
await request('https://example.com', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
content: JSON.stringify({
a: 'hello',
b: 'world',
}),
});
It would make a HTTP request like:
POST / HTTP/1.1
host: example.com
content-type: application/json
{
"a": "hello",
"b": "world"
}
This exmaple can use options.data with application/json content type:
await request('https://example.com', {
method: 'POST',
headers: {
'content-type': 'application/json'
},
data: {
a: 'hello',
b: 'world',
}
});
Options: options.files
Upload a file with a hello field.
await request('https://example.com/upload', {
method: 'POST',
files: __filename,
data: {
hello: 'hello urllib',
},
});
Upload multi files with a hello field.
await request('https://example.com/upload', {
method: 'POST',
files: [
__filename,
fs.createReadStream(__filename),
Buffer.from('mock file content'),
],
data: {
hello: 'hello urllib with multi files',
},
});
Custom file field name with uploadfile.
await request('https://example.com/upload', {
method: 'POST',
files: {
uploadfile: __filename,
},
});
Response Object
Response is normal object, it contains:
statusorstatusCode: response status code.-1meaning some network error likeENOTFOUND-2meaning ConnectionTimeoutError
headers: response http headers, default is{}size: response sizeaborted: response was aborted or notrt: total request and response time in ms.timing: timing object if timing enable.socket: socket info
Run test with debug log
NODE_DEBUG=urllib:* npm test
Mocking Request
export from undici
import { strict as assert } from 'assert';
import { MockAgent, setGlobalDispatcher, request } from 'urllib';
const mockAgent = new MockAgent();
setGlobalDispatcher(mockAgent);
const mockPool = mockAgent.get('http://localhost:7001');
mockPool.intercept({
path: '/foo',
method: 'POST',
}).reply(400, {
message: 'mock 400 bad request',
});
const response = await request('http://localhost:7001/foo', {
method: 'POST',
dataType: 'json',
});
assert.equal(response.status, 400);
assert.deepEqual(response.data, { message: 'mock 400 bad request' });
Request through a http proxy
export from undici
import { ProxyAgent, request } from 'urllib';
const proxyAgent = new ProxyAgent('http://my.proxy.com:8080');
const response = await request('https://www.npmjs.com/package/urllib', {
dispatcher: proxyAgent,
});
console.log(response.status, response.headers);
Benchmarks
Connections 1
| Tests | Samples | Result | Tolerance | Difference with slowest |
|---|---|---|---|---|
| http - no keepalive | 15 | 6.38 req/sec | ± 2.44 % | - |
| http - keepalive | 10 | 6.77 req/sec | ± 2.35 % | + 6.13 % |
| urllib2 - request | 45 | 40.13 req/sec | ± 2.88 % | + 528.66 % |
| urllib3 - request | 10 | 58.51 req/sec | ± 2.52 % | + 816.64 % |
| undici - pipeline | 5 | 59.12 req/sec | ± 2.47 % | + 826.18 % |
| undici - fetch | 15 | 60.42 req/sec | ± 3.00 % | + 846.60 % |
| undici - dispatch | 5 | 60.58 req/sec | ± 1.39 % | + 848.99 % |
| undici - stream | 5 | 61.30 req/sec | ± 1.31 % | + 860.39 % |
| undici - request | 5 | 61.74 req/sec | ± 2.03 % | + 867.20 % |
Connections 50
| Tests | Samples | Result | Tolerance | Difference with slowest |
|---|---|---|---|---|
| urllib2 - request | 51 | 1465.40 req/sec | ± 14.40 % | - |
| undici - fetch | 40 | 3121.10 req/sec | ± 2.82 % | + 112.99 % |
| http - no keepalive | 45 | 3355.42 req/sec | ± 2.84 % | + 128.98 % |
| http - keepalive | 51 | 5179.55 req/sec | ± 36.61 % | + 253.46 % |
| urllib3 - request | 30 | 7045.86 req/sec | ± 2.93 % | + 380.82 % |
| undici - pipeline | 50 | 8306.92 req/sec | ± 2.99 % | + 466.87 % |
| undici - request | 51 | 9552.59 req/sec | ± 13.13 % | + 551.88 % |
| undici - stream | 45 | 12523.45 req/sec | ± 2.97 % | + 754.61 % |
| undici - dispatch | 51 | 12970.18 req/sec | ± 3.15 % | + 785.10 % |
Contributors
This project follows the git-contributor spec, auto updated at Mon Dec 04 2023 00:13:39 GMT+0800.