ido-plutoiPull
> Super fast file downloader with multiple connections
bash
npx ipull http://example.com/file.large
## Features
- Download using parallels connections
- Maximize download speed (automatic parallelization, 3+)
- Pausing and resuming downloads
- Node.js and browser support
- Smart retry on fail
- CLI Progress bar
- Download statistics (speed, time left, etc.)
### NodeJS API
ts
import {downloadFile} from 'ipull';
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path', // or 'savePath' for full path
cliProgress: true, // Show progress bar in the CLI (default: false)
parallelStreams: 3 // Number of parallel connections (default: 3)
});
await downloader.download();
## Browser support
Download a file in the browser using multiple connections
ts
import {downloadFileBrowser} from "ipull/dist/browser.js";
const downloader = await downloadFileBrowser({
url: 'https://example.com/file.large',
acceptRangeIsKnown: true, // overcome CORS, force multi-connection download (use only if you know the server supports range requests)
// defaultFetchDownloadInfo: { // set download info manually to overcome CORS issues && prevent multiple requests
// acceptRange: true,
// length: 40789822,
// }
});
await downloader.download();
image.src = downloader.writeStream.resultAsBlobURL();
console.log(downloader.writeStream.result); // Uint8Array
### Custom stream
You can use a custom stream
ts
import {downloadFileBrowser} from "ipull/dist/browser.js";
const downloader = await downloadFileBrowser({
url: 'https://example.com/file.large',
onWrite: (cursor: number, buffers: Uint8Array[], options) => {
const totalLength = buffers.reduce((acc, buffer) => acc + buffer.length, 0);
console.log(`Writing ${totalLength} bytes at cursor ${cursor}, with options: ${JSON.stringify(options)}`);
}
});
await downloader.download();
console.log(downloader.writeStream.result.length === 0); // true, because we write to a custom stream
## CLI
Usage: ipull [options] [files...]
Pull/copy files from a remote server/local directory
Arguments:
files Files to pull/copy
Options:
-s --save [path] Save location (directory/file)
-c --connections [number] Number of parallel connections (default: "4")
-p --program [type] The download strategy (choices: "stream", "chunks")
-t --truncate-name Truncate file names in the CLI status to make them appear shorter
-V, --version output the version number
-h, --help display help for command
Commands:
set [options] [path] <value> Set download locations
### Set custom save directory
You can set a custom save directory by using the set command.
bash
ipull set .zip ~/Downloads/zips
(use default to set the default save directory)
## Advanced usage
### Skip existing files
Skip downloading files that already exist in the save location and have the same size.
ts
import {downloadFile} from 'ipull';
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path',
skipExisting: true
});
### Download file from parts
Consolidate multiple files parts into one file.
Beneficial for downloading large files from servers that limit file size. (e.g. HuggingFace models)
ts
import {downloadFile} from 'ipull';
const downloadParts = [
"https://example.com/file.large-part-1",
"https://example.com/file.large-part-2",
"https://example.com/file.large-part-3",
];
const downloader = await downloadFile({
partURLs: downloadParts,
directory: './this/path',
filename: 'file.large'
});
await downloader.download();
The split must be binary and not a zip-split
### Custom headers
You can set custom headers for the download request
ts
import {downloadFile} from 'ipull';
const downloader = await downloadFile({
url: 'https://example.com/file.large',
savePath: './this/path/file.large',
headers: {
'Authorization': 'Bearer token 1'
},
// You can also add alternative headers in case of an 400-499 error
tryHeaders: [
{
Authorization: 'Bearer token 2'
}
]
});
await downloader.download();
### Abort download
You can cancel the download by calling the close method (it will not delete the file).
If you want to also delete the file, you can call the closeAndDeleteFile method.
ts
import {downloadFile} from 'ipull';
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path'
});
setTimeout(() => {
downloader.close();
}, 5_000);
await downloader.download();
### Pause & Resume download
ts
import {downloadFile} from 'ipull';
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path'
});
setInterval(() => {
downloader.pause();
setTimeout(() => {
downloader.resume();
}, 5_000);
}, 10_000);
await downloader.download();
The pause may take a few seconds to actually pause the download, because it waits for the current connections to
finish
### Error handling
If a network/file-system error occurs, the download will automatically retry
with async-retry
If the maximum reties was reached the download will fail and an error will be thrown from the download() call:
ts
import {downloadFile} from 'ipull';
try {
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path'
});
await downloader.download();
} catch (error) {
console.error(`Download failed: ${error.message}`);
}
### Download Stuck
In some edge cases, the re-try mechanism may give the illusion that the download is stuck.
(You can see this in the progress object that "retrying" is true)
To debug this, disable the re-try mechanism:
js
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path',
retry: {
retries: 0
},
retryFetchDownloadInfo: {
retries: 0
}
});
### Listening to events
Events are emitted using the EventEmitter pattern and can be listened to using the on method
ts
interface DownloadEngineEvents {
start: [];
paused: [];
resumed: [];
progress: [FormattedStatus];
save: [DownloadProgressInfo];
finished: [];
closed: [];
}
const downloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path'
});
downloader.on("progress", (progress) => {
console.log(`Downloaded ${progress.transferred} bytes`);
});
### Remote Download Listing
If you want to show in the CLI the progress of a file downloading in remote.
ts
const originaldownloader = await downloadFile({
url: 'https://example.com/file.large',
directory: './this/path'
});
const remoteDownloader = downloadFileRemote({
cliProgress: true
});
originaldownloader.on("progress", (progress) => {
remoteDownloader.emitRemoteProgress(progress);
});
await originaldownloader.download();
### Download multiple files
If you want to download multiple files, you can use the downloadSequence function.
By default, it will download files one by one, but you can set the parallel option to download them in parallel.
It is better to download one file at a time if you are downloading from the same server (as it may limit the number of
connections).
ts
import {downloadFile, downloadSequence} from "ipull";
const downloader = await downloadSequence(
{
cliProgress: true,
// parallelDownloads: 2, download 2 files in parallel, default is 1
},
downloadFile({
url: "https://example.com/file1.large",
directory: "."
}),
downloadFile({
url: "https://example.com/file2.large",
directory: "."
}),
);
console.log(`Downloading ${downloader.downloads.length} files...`);
await downloader.download();
### Custom progress bar
ts
import {downloadFile, FormattedStatus} from "ipull";
function createProgressBar({fileName, ...data}: FormattedStatus) {
return `${fileName} ${JSON.stringify(data)}`;
}
const downloader = await downloadFile({
url: "https://example.com/file.large",
directory: "./this/path",
cliStyle: createProgressBar
});
await downloader.download();
If you like this repo, star it ✨