node-modulesurllib
Request HTTP URLs in a complex world — basic and digest authentication, redirections, timeout and more.
Install
npm install urllib
Usage
TypeScript and ESM
import { request } from 'urllib';
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
CommonJS
const { request } = require('urllib');
const { data, res } = await request('http://cnodejs.org/');
// result: { data: Buffer, res: Response }
console.log('status: %s, body size: %d, headers: %j', res.status, data.length, res.headers);
API Doc
Method: async request(url[, options])
Arguments
- url String | Object - The URL to request, either a String or a Object that return by url.parse.
- options Object - Optional
- method String - Request method, defaults to
GET. Could beGET,POST,DELETEorPUT. Alias 'type'. - data Object - Data to be sent. Will be stringify automatically.
- content String | Buffer - Manually set the content of payload. If set,
datawill be ignored. - stream stream.Readable - Stream to be pipe to the remote. If set,
dataandcontentwill be ignored. - writeStream stream.Writable - A writable stream to be piped by the response stream. Responding data will be write to this stream and
callbackwill be called withdatasetnullafter finished writing. - files {Array<ReadStream|Buffer|String> | Object | ReadStream | Buffer | String - The files will send with
multipart/form-dataformat, base onformstream. Ifmethodnot set, will usePOSTmethod by default. - contentType String - Type of request data. Could be
json(Notes: not useapplication/jsonhere). If it'sjson, will auto setContent-Type: application/jsonheader. - dataType String - Type of response data. Could be
textorjson. If it'stext, thecallbackeddatawould be a String. If it'sjson, thedataof callback would be a parsed JSON Object and will auto setAccept: application/jsonheader. Defaultcallbackeddatawould be aBuffer. - fixJSONCtlChars Boolean - Fix the control characters (U+0000 through U+001F) before JSON parse response. Default is
false. - headers Object - Request headers.
- timeout Number | Array - Request timeout in milliseconds for connecting phase and response receiving phase. Default is
5000. You can usetimeout: 5000to tell urllib use same timeout on two phase or set them seperately such astimeout: [3000, 5000], which will set connecting timeout to 3s and response 5s. - keepAliveTimeout
number | null- Default is4000, 4 seconds - The timeout after which a socket without active requests will time out. Monitors time between activity on a connected socket. This value may be overridden by keep-alive hints from the server. See MDN: HTTP - Headers - Keep-Alive directives for more details. - auth String -
username:passwordused in HTTP Basic Authorization. - digestAuth String -
username:passwordused in HTTP Digest Authorization. - followRedirect Boolean - follow HTTP 3xx responses as redirects. defaults to true.
- maxRedirects Number - The maximum number of redirects to follow, defaults to 10.
- formatRedirectUrl Function - Format the redirect url by your self. Default is
url.resolve(from, to). - beforeRequest Function - Before request hook, you can change every thing here.
- streaming Boolean - let you get the
resobject when request connected, defaultfalse. aliascustomResponse - compressed Boolean - Accept
gzip, brresponse content and auto decode it, default isfalse. - timing Boolean - Enable timing or not, default is
true. - socketPath String | null - request a unix socket service, default is
null. - highWaterMark Number - default is
67108864, 64 KiB.
- method String - Request method, defaults to
Options: options.data
When making a request:
await request('https://example.com', {
method: 'GET',
data: {
'a': 'hello',
'b': 'world',
},
});
For GET request, data will be stringify to query string, e.g. http://example.com/?a=hello&b=world.
For others like POST, PATCH or PUT request,
in defaults, the data will be stringify into application/x-www-form-urlencoded format
if content-type header is not set.
If content-type is application/json, the data will be JSON.stringify to JSON data format.
Options: options.content
options.content is useful when you wish to construct the request body by yourself,
for example making a content-type: application/json request.
Notes that if you want to send a JSON body, you should stringify it yourself:
await request('https://example.com', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
content: JSON.stringify({
a: 'hello',
b: 'world',
}),
});
It would make a HTTP request like:
POST / HTTP/1.1
host: example.com
content-type: application/json
{
"a": "hello",
"b": "world"
}
This exmaple can use options.data with application/json content type:
await request('https://example.com', {
method: 'POST',
headers: {
'content-type': 'application/json'
},
data: {
a: 'hello',
b: 'world',
}
});
Options: options.files
Upload a file with a hello field.
await request('https://example.com/upload', {
method: 'POST',
files: __filename,
data: {
hello: 'hello urllib',
},
});
Upload multi files with a hello field.
await request('https://example.com/upload', {
method: 'POST',
files: [
__filename,
fs.createReadStream(__filename),
Buffer.from('mock file content'),
],
data: {
hello: 'hello urllib with multi files',
},
});
Custom file field name with uploadfile.
await request('https://example.com/upload', {
method: 'POST',
files: {
uploadfile: __filename,
},
});
Response Object
Response is normal object, it contains:
statusorstatusCode: response status code.-1meaning some network error likeENOTFOUND-2meaning ConnectionTimeoutError
headers: response http headers, default is{}size: response sizeaborted: response was aborted or notrt: total request and response time in ms.timing: timing object if timing enable.socket: socket info
Run test with debug log
NODE_DEBUG=urllib:* npm test
Request with HTTP2
Create a HttpClient with options.allowH2 = true
import { HttpClient } from 'urllib';
const httpClient = new HttpClient({
allowH2: true,
});
const response = await httpClient.request('https://node.js.org');
console.log(response.status);
console.log(response.headers);
Mocking Request
export from undici
import { strict as assert } from 'assert';
import { MockAgent, setGlobalDispatcher, request } from 'urllib';
const mockAgent = new MockAgent();
setGlobalDispatcher(mockAgent);
const mockPool = mockAgent.get('http://localhost:7001');
mockPool.intercept({
path: '/foo',
method: 'POST',
}).reply(400, {
message: 'mock 400 bad request',
});
const response = await request('http://localhost:7001/foo', {
method: 'POST',
dataType: 'json',
});
assert.equal(response.status, 400);
assert.deepEqual(response.data, { message: 'mock 400 bad request' });
Request through a http proxy
export from undici
import { ProxyAgent, request } from 'urllib';
const proxyAgent = new ProxyAgent('http://my.proxy.com:8080');
const response = await request('https://www.npmjs.com/package/urllib', {
dispatcher: proxyAgent,
});
console.log(response.status, response.headers);
Benchmarks
undici@6.19.2
Node.js v18.20.3
┌─────────┬───────────────────────┬─────────┬────────────────────┬─────────────┬─────────────────────────┐
│ (index) │ Tests │ Samples │ Result │ Tolerance │ Difference with slowest │
├─────────┼───────────────────────┼─────────┼────────────────────┼─────────────┼─────────────────────────┤
│ 0 │ 'urllib2 - request' │ 10 │ '321.53 req/sec' │ '± 0.38 %' │ '-' │
│ 1 │ 'http - no keepalive' │ 10 │ '607.77 req/sec' │ '± 0.80 %' │ '+ 89.02 %' │
│ 2 │ 'got' │ 101 │ '7929.51 req/sec' │ '± 4.46 %' │ '+ 2366.15 %' │
│ 3 │ 'node-fetch' │ 40 │ '8651.95 req/sec' │ '± 2.99 %' │ '+ 2590.84 %' │
│ 4 │ 'request' │ 101 │ '8864.09 req/sec' │ '± 7.81 %' │ '+ 2656.82 %' │
│ 5 │ 'undici - fetch' │ 101 │ '9607.01 req/sec' │ '± 4.23 %' │ '+ 2887.87 %' │
│ 6 │ 'axios' │ 55 │ '10378.80 req/sec' │ '± 2.94 %' │ '+ 3127.91 %' │
│ 7 │ 'superagent' │ 75 │ '11286.74 req/sec' │ '± 2.90 %' │ '+ 3410.29 %' │
│ 8 │ 'http - keepalive' │ 60 │ '11288.96 req/sec' │ '± 2.95 %' │ '+ 3410.98 %' │
│ 9 │ 'urllib4 - request' │ 101 │ '11352.65 req/sec' │ '± 10.20 %' │ '+ 3430.79 %' │
│ 10 │ 'urllib3 - request' │ 40 │ '13831.19 req/sec' │ '± 2.89 %' │ '+ 4201.64 %' │
│ 11 │ 'undici - pipeline' │ 60 │ '14562.44 req/sec' │ '± 2.91 %' │ '+ 4429.06 %' │
│ 12 │ 'undici - request' │ 70 │ '19630.64 req/sec' │ '± 2.87 %' │ '+ 6005.32 %' │
│ 13 │ 'undici - stream' │ 55 │ '20843.50 req/sec' │ '± 2.90 %' │ '+ 6382.54 %' │
│ 14 │ 'undici - dispatch' │ 55 │ '21233.10 req/sec' │ '± 2.82 %' │ '+ 6503.70 %' │
└─────────┴───────────────────────┴─────────┴────────────────────┴─────────────┴─────────────────────────┘
Node.js v20.15.0
┌─────────┬───────────────────────┬─────────┬────────────────────┬────────────┬─────────────────────────┐
│ (index) │ Tests │ Samples │ Result │ Tolerance │ Difference with slowest │
├─────────┼───────────────────────┼─────────┼────────────────────┼────────────┼─────────────────────────┤
│ 0 │ 'urllib2 - request' │ 10 │ '332.91 req/sec' │ '± 1.13 %' │ '-' │
│ 1 │ 'http - no keepalive' │ 10 │ '615.50 req/sec' │ '± 2.25 %' │ '+ 84.88 %' │
│ 2 │ 'got' │ 55 │ '7658.39 req/sec' │ '± 2.98 %' │ '+ 2200.42 %' │
│ 3 │ 'node-fetch' │ 30 │ '7832.96 req/sec' │ '± 2.96 %' │ '+ 2252.86 %' │
│ 4 │ 'axios' │ 40 │ '8607.27 req/sec' │ '± 2.79 %' │ '+ 2485.44 %' │
│ 5 │ 'request' │ 35 │ '8703.49 req/sec' │ '± 2.84 %' │ '+ 2514.35 %' │
│ 6 │ 'undici - fetch' │ 65 │ '9971.24 req/sec' │ '± 2.96 %' │ '+ 2895.15 %' │
│ 7 │ 'superagent' │ 30 │ '11006.46 req/sec' │ '± 2.90 %' │ '+ 3206.11 %' │
│ 8 │ 'http - keepalive' │ 55 │ '11610.14 req/sec' │ '± 2.87 %' │ '+ 3387.44 %' │
│ 9 │ 'urllib3 - request' │ 25 │ '13873.38 req/sec' │ '± 2.96 %' │ '+ 4067.27 %' │
│ 10 │ 'urllib4 - request' │ 25 │ '14291.36 req/sec' │ '± 2.92 %' │ '+ 4192.82 %' │
│ 11 │ 'undici - pipeline' │ 45 │ '14617.69 req/sec' │ '± 2.84 %' │ '+ 4290.85 %' │
│ 12 │ 'undici - dispatch' │ 101 │ '18716.29 req/sec' │ '± 3.97 %' │ '+ 5521.98 %' │
│ 13 │ 'undici - request' │ 101 │ '19165.16 req/sec' │ '± 3.25 %' │ '+ 5656.81 %' │
│ 14 │ 'undici - stream' │ 30 │ '21816.28 req/sec' │ '± 2.99 %' │ '+ 6453.15 %' │
└─────────┴───────────────────────┴─────────┴────────────────────┴────────────┴─────────────────────────┘
Node.js v22.3.0
┌─────────┬───────────────────────┬─────────┬────────────────────┬────────────┬─────────────────────────┐
│ (index) │ Tests │ Samples │ Result │ Tolerance │ Difference with slowest │
├─────────┼───────────────────────┼─────────┼────────────────────┼────────────┼─────────────────────────┤
│ 0 │ 'urllib2 - request' │ 15 │ '297.46 req/sec' │ '± 2.65 %' │ '-' │
│ 1 │ 'http - no keepalive' │ 10 │ '598.25 req/sec' │ '± 1.94 %' │ '+ 101.12 %' │
│ 2 │ 'axios' │ 30 │ '8487.94 req/sec' │ '± 2.91 %' │ '+ 2753.52 %' │
│ 3 │ 'got' │ 50 │ '10054.46 req/sec' │ '± 2.89 %' │ '+ 3280.16 %' │
│ 4 │ 'request' │ 45 │ '10306.02 req/sec' │ '± 2.87 %' │ '+ 3364.73 %' │
│ 5 │ 'node-fetch' │ 55 │ '11160.02 req/sec' │ '± 2.87 %' │ '+ 3651.83 %' │
│ 6 │ 'superagent' │ 80 │ '11302.28 req/sec' │ '± 2.85 %' │ '+ 3699.66 %' │
│ 7 │ 'undici - fetch' │ 60 │ '11357.87 req/sec' │ '± 2.89 %' │ '+ 3718.35 %' │
│ 8 │ 'http - keepalive' │ 60 │ '13782.10 req/sec' │ '± 2.97 %' │ '+ 4533.34 %' │
│ 9 │ 'urllib4 - request' │ 70 │ '15965.62 req/sec' │ '± 2.88 %' │ '+ 5267.40 %' │
│ 10 │ 'urllib3 - request' │ 55 │ '16010.37 req/sec' │ '± 2.90 %' │ '+ 5282.45 %' │
│ 11 │ 'undici - pipeline' │ 35 │ '17969.37 req/sec' │ '± 2.95 %' │ '+ 5941.03 %' │
│ 12 │ 'undici - dispatch' │ 101 │ '18765.50 req/sec' │ '± 3.01 %' │ '+ 6208.68 %' │
│ 13 │ 'undici - request' │ 85 │ '20091.12 req/sec' │ '± 2.95 %' │ '+ 6654.33 %' │
│ 14 │ 'undici - stream' │ 45 │ '21599.12 req/sec' │ '± 2.81 %' │ '+ 7161.30 %' │
└─────────┴───────────────────────┴─────────┴────────────────────┴────────────┴─────────────────────────┘
License
Contributors
Made with contributors-img.