前端大文件上传解决方案
本文分享的内容是前端大文件上传的解决方案，文件上传是前端开发中常见的需求，特别是在处理视频、大型文档或数据集时。对于小文件上传不做详细介绍，在源码中已附带。
大文件上传前置条件
- 设置分片大小的值，即规定每个切片的大小
- 设置文件大小阈值，即超过多少M判定为大文件
大文件上传步骤
- 计算文件md5的值
- 前端对文件进行分割，每个切片中包含
索引、切片内容、文件名称 - 对切片集合进行遍历，按照顺序上传切片
- 先校验切片是否已上传，若是则直接进入下一个切片的上传，否则进行上传操作
- 重复上一步，直至所有切片都已完成上传
- 调用
合并接口，对切片进行合并
优化点：
- 源码中添加切片上传的并发控制
- 自动重试机制（每个分片最多重试3次）
重点代码解释
-
文件分片处理
// 创建文件分片数据 const createFileChunks = (file) => { const chunks = []; let cur = 0; while (cur < file.size) { chunks.push({ fileName: file.name, chunkNumber: chunks.length + 1, file: file.slice(cur, cur + pageInfo.chunkSize), }); cur += pageInfo.chunkSize; } return chunks; };- 将大文件分割为3MB大小的分片
- 每个分片包含文件名、分片序号和分片数据
- 使用
file.slice方法进行文件分割
-
MD5计算
// 计算文件md5的值 const calculateFileMD5 = (file) => { return new Promise((resolve) => { const spark = new SparkMD5.ArrayBuffer(); const fileReader = new FileReader(); const totalChunks = Math.ceil(file.size / pageInfo.chunkSize); let currentChunk = 0; const loadNextChunk = () => { const start = currentChunk * pageInfo.chunkSize; const end = Math.min(start + pageInfo.chunkSize, file.size); fileReader.readAsArrayBuffer(file.raw.slice(start, end)); }; fileReader.onload = (e) => { spark.append(e.target.result); currentChunk++; if (currentChunk < totalChunks) { loadNextChunk(); } else { pageInfo.md5 = spark.end(); resolve(state.md5); } }; loadNextChunk(); }) };- 使用SparkMD5库计算文件MD5值
- 分片读取文件内容，避免一次性加载大文件导致内存问题
- 增量计算MD5，最终合并得到完整文件的MD5
-
并发数量控制
const pageInfo = reactive({ ..., concurrency: 3, // 并发数 }); const uploadChunks = async (chunks) => { try { const results = []; const executing = new Set(); // 正在执行的 for (const chunk of chunks) { // 如果达到最大并发数，等待一个完成 if (executing.size >= pageInfo.concurrency) { await Promise.race(executing); } // 创建并跟踪请求 const promise = processChunk(chunk).then(result => { executing.delete(promise); return result; }); executing.add(promise); results.push(promise); } // 等待所有剩余请求完成 return await Promise.all(results); } catch (error) { return false; } };- 使用 Set 跟踪正在执行的请求
- 通过 Promise.race 实现并发限制
- 最大并发数由 pageInfo.concurrency 控制（默认为3）
-
重试机制
// 每个分片最多重试3次，对应retries参数的值 const processChunk = async (chunk, retries = 3) => { for (let i = 0; i < retries; i++) { try { // 先检查是否已上传 const { data } = await checkChunkUploadStatus(chunk); if (!data) { console.log(`分片 ${chunk.chunkNumber} 未上传，开始上传 (尝试 ${i + 1}/${retries})`); return await chunkUploadF(chunk); } console.log(`分片 ${chunk.chunkNumber} 已存在，跳过上传`); return { code: 0, data: true, message: "成功", resultMsg: null, chunkNumber: chunk.chunkNumber, }; } catch (error) { console.error(`分片 ${chunk.chunkNumber} 上传失败 (尝试 ${i + 1}/${retries}):`, error); if (i === retries - 1) throw error; await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1))); // 指数退避 } } }; -
指数退避策略
await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1))); // 指数退避- 第一次重试等待：1秒 (1000 * 1)
- 第二次重试等待：2秒 (1000 * 2)
- 第三次重试等待：3秒 (1000 * 3)
- 这种逐渐增加等待时间的方式称为"指数退避"，可以有效避免网络拥塞
-
文件上传入口
// 文件上传主函数 const uploadFileF = async () => { pageInfo.fileLoading = true; if (!pageInfo.isOverThreshold) { // 小文件直接上传 const formData = new FormData(); formData.append("file", pageInfo.formInfo.fileRaw); // ...省略上传代码 } else { // 大文件分片上传 const fileChunkList = createFileChunks(pageInfo.formInfo.fileRaw); const res = await uploadChunks(fileChunkList); if (res) { // 检查所有分片是否上传成功 const flag = res.every((ele) => ele.data); if (!flag) { proxy.$message.error("存在上传失败的分片，请重新上传失败"); } } // 合并分片 const { code, data, message } = await mergeChunks( pageInfo.formInfo.fileName, fileChunkList.length ); // 处理合并结果 } };- 根据文件大小选择不同上传策略（5MB为阈值）
- 小文件直接上传，大文件走分片上传流程
- 最终合并分片完成上传
本文源码中并未增加进度设计，如有需要可以自行添加。根据上传的切片数量计算即可。其次在最后我添加了测试文件，该文件可以用于前端在开发前的测试和功能集成。
使用
npm install spark-md5
# 或者
yarn add spark-md5
源码
点击上传
测试文件
DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>文件分片上传测试工具title>
<script src="https://cdn.tailwindcss.com">script>
<link href="https://cdn.jsdelivr.net/npm/[email protected]/css/font-awesome.min.css" rel="stylesheet">
<script src="https://cdnjs.cloudflare.com/ajax/libs/spark-md5/3.0.2/spark-md5.min.js">script>
<script>
tailwind.config = {
theme: {
extend: {
colors: {
primary: '#165DFF',
secondary: '#0A2463',
accent: '#3E92CC',
dark: '#050A30',
light: '#E8F1F2',
success: '#36D399',
warning: '#FFAB00',
error: '#F87272'
},
fontFamily: {
inter: ['Inter', 'system-ui', 'sans-serif'],
},
}
}
}
script>
<style type="text/tailwindcss">
@layer utilities {
.content-auto {
content-visibility: auto;
}
.upload-drop-area {
@apply border-2 border-dashed border-primary/30 rounded-lg p-8 text-center transition-all duration-300 hover:border-primary/60 hover:bg-primary/5;
}
.upload-drop-area-active {
@apply border-primary bg-primary/10;
}
.progress-bar {
@apply h-2 bg-gray-200 rounded-full overflow-hidden;
}
.progress-value {
@apply h-full bg-primary transition-all duration-300 ease-out;
}
.btn-primary {
@apply bg-primary hover:bg-primary/90 text-white font-medium py-2 px-6 rounded-lg transition-all duration-300 shadow-lg hover:shadow-xl transform hover:-translate-y-0.5;
}
.btn-secondary {
@apply bg-gray-200 hover:bg-gray-300 text-gray-800 font-medium py-2 px-6 rounded-lg transition-all duration-300;
}
.file-item {
@apply bg-white rounded-lg shadow-md p-4 mb-4 flex items-center justify-between transition-all duration-300 hover:shadow-lg;
}
.file-icon {
@apply w-12 h-12 flex items-center justify-center rounded-lg mr-4 text-2xl;
}
.file-info {
@apply flex-1 min-w-0;
}
.file-name {
@apply font-medium text-gray-900 truncate;
}
.file-size {
@apply text-sm text-gray-500;
}
.upload-status {
@apply flex items-center text-sm;
}
.fade-in {
animation: fadeIn 0.5s ease-in-out;
}
@keyframes fadeIn {
from { opacity: 0; transform: translateY(10px); }
to { opacity: 1; transform: translateY(0); }
}
.pulse {
animation: pulse 2s infinite;
}
@keyframes pulse {
0% { opacity: 1; }
50% { opacity: 0.5; }
100% { opacity: 1; }
}
}
style>
head>
<body class="bg-gray-50 font-inter text-gray-800 min-h-screen">
<div class="container mx-auto px-4 py-8 max-w-6xl">
<header class="mb-8 text-center">
<h1 class="text-[clamp(1.8rem,4vw,2.5rem)] font-bold text-dark mb-2">
<i class="fa fa-cloud-upload text-primary mr-2">i>文件分片上传测试工具
h1>
<p class="text-gray-600 max-w-2xl mx-auto">支持大文件分片上传、断点续传和MD5校验，可自定义分片大小和并发数p>
header>
<main class="bg-white rounded-xl shadow-xl overflow-hidden">
<div class="p-6 border-b border-gray-200 bg-gray-50">
<h2 class="text-xl font-semibold mb-4 flex items-center">
<i class="fa fa-sliders text-primary mr-2">i>上传配置
h2>
<div class="grid grid-cols-1 md:grid-cols-3 gap-4">
<div>
<label class="block text-sm font-medium text-gray-700 mb-1">分片大小label>
<div class="flex items-center">
<input type="number" id="chunkSize" value="5" min="1" max="100"
class="w-full rounded-l-lg border border-gray-300 py-2 px-3 focus:outline-none focus:ring-2 focus:ring-primary/50 focus:border-primary">
<span class="bg-gray-100 border border-l-0 border-gray-300 rounded-r-lg px-3 py-2 text-gray-700">MBspan>
div>
div>
<div>
<label class="block text-sm font-medium text-gray-700 mb-1">并发数label>
<input type="number" id="concurrency" value="3" min="1" max="10"
class="w-full rounded-lg border border-gray-300 py-2 px-3 focus:outline-none focus:ring-2 focus:ring-primary/50 focus:border-primary">
div>
<div>
<label class="block text-sm font-medium text-gray-700 mb-1">服务器URLlabel>
<input type="url" id="uploadUrl" value="/api/upload/chunk"
class="w-full rounded-lg border border-gray-300 py-2 px-3 focus:outline-none focus:ring-2 focus:ring-primary/50 focus:border-primary">
div>
div>
div>
<div class="p-8">
<div id="dropArea" class="upload-drop-area mb-8">
<div class="space-y-4">
<i class="fa fa-cloud-upload text-5xl text-primary/60">i>
<h3 class="text-xl font-semibold text-gray-800">拖放文件到此处上传h3>
<p class="text-gray-500">或者p>
<label for="fileInput" class="btn-primary inline-flex items-center">
<i class="fa fa-file-text-o mr-2">i>选择文件
<input type="file" id="fileInput" class="hidden" multiple>
label>
<p class="text-sm text-gray-400">支持多文件上传，最大文件大小无限制p>
div>
div>
<div>
<h2 class="text-xl font-semibold mb-4 flex items-center">
<i class="fa fa-file-o text-primary mr-2">i>文件列表
h2>
<div id="fileList" class="space-y-3">div>
div>
div>
main>
<footer class="mt-12 text-center text-gray-500 text-sm">
<p>© 2023 文件分片上传测试工具 | 支持断点续传和MD5校验p>
footer>
div>
<script>
// 文件上传管理器
class FileUploadManager {
constructor() {
this.files = [];
this.chunkSize = 5 * 1024 * 1024; // 默认5MB
this.concurrency = 3; // 默认并发数
this.uploadUrl = '/api/upload/chunk';
this.initEventListeners();
}
// 初始化事件监听
initEventListeners() {
// 文件选择
document.getElementById('fileInput').addEventListener('change', e => {
this.handleFiles(e.target.files);
});
// 拖放事件
const dropArea = document.getElementById('dropArea');
['dragenter', 'dragover', 'dragleave', 'drop'].forEach(eventName => {
dropArea.addEventListener(eventName, this.preventDefaults, false);
});
['dragenter', 'dragover'].forEach(eventName => {
dropArea.addEventListener(eventName, () => {
dropArea.classList.add('upload-drop-area-active');
}, false);
});
['dragleave', 'drop'].forEach(eventName => {
dropArea.addEventListener(eventName, () => {
dropArea.classList.remove('upload-drop-area-active');
}, false);
});
dropArea.addEventListener('drop', e => {
this.handleFiles(e.dataTransfer.files);
}, false);
// 配置更改
document.getElementById('chunkSize').addEventListener('change', e => {
this.chunkSize = parseInt(e.target.value) * 1024 * 1024;
});
document.getElementById('concurrency').addEventListener('change', e => {
this.concurrency = parseInt(e.target.value);
});
document.getElementById('uploadUrl').addEventListener('change', e => {
this.uploadUrl = e.target.value;
});
}
// 阻止默认事件
preventDefaults(e) {
e.preventDefault();
e.stopPropagation();
}
// 处理选择的文件
handleFiles(files) {
if (!files.length) return;
Array.from(files).forEach(file => {
if (this.isFileAdded(file)) return;
const fileItem = this.createFileItem(file);
document.getElementById('fileList').appendChild(fileItem);
this.files.push({
file,
element: fileItem,
status: 'ready',
progress: 0,
md5: null,
chunks: []
});
// 计算文件MD5
this.calculateFileMD5(file, fileItem);
});
}
// 检查文件是否已添加
isFileAdded(file) {
return this.files.some(item => item.file.name === file.name && item.file.size === file.size);
}
// 创建文件项DOM
createFileItem(file) {
const fileType = this.getFileType(file.name);
const fileSize = this.formatFileSize(file.size);
const div = document.createElement('div');
div.className = 'file-item fade-in';
div.innerHTML = `
${file.name}
${fileSize}
准备中...
`;
// 添加事件监听
div.querySelector('.upload-btn').addEventListener('click', () => {
this.startUpload(file);
});
div.querySelector('.cancel-btn').addEventListener('click', () => {
this.cancelUpload(file);
});
return div;
}
// 获取文件类型
getFileType(fileName) {
const ext = fileName.split('.').pop().toLowerCase();
const imageExts = ['jpg', 'jpeg', 'png', 'gif', 'bmp', 'webp'];
const videoExts = ['mp4', 'avi', 'mov', 'mkv', 'wmv'];
const audioExts = ['mp3', 'wav', 'ogg', 'flac'];
const docExts = ['doc', 'docx', 'pdf', 'txt', 'ppt', 'pptx', 'xls', 'xlsx'];
if (imageExts.includes(ext)) return 'image';
if (videoExts.includes(ext)) return 'video';
if (audioExts.includes(ext)) return 'audio';
if (docExts.includes(ext)) return 'document';
return 'unknown';
}
// 获取文件图标
getFileIcon(fileType) {
const icons = {
'image': 'fa-file-image-o',
'video': 'fa-file-video-o',
'audio': 'fa-file-audio-o',
'document': 'fa-file-text-o',
'unknown': 'fa-file-o'
};
return icons[fileType] || 'fa-file-o';
}
// 获取文件图标颜色类
getFileIconClass(fileType) {
const colors = {
'image': 'bg-blue-100 text-blue-600',
'video': 'bg-red-100 text-red-600',
'audio': 'bg-green-100 text-green-600',
'document': 'bg-yellow-100 text-yellow-600',
'unknown': 'bg-gray-100 text-gray-600'
};
return colors[fileType] || 'bg-gray-100 text-gray-600';
}
// 格式化文件大小
formatFileSize(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
// 计算文件MD5
calculateFileMD5(file, fileItem) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const statusText = fileItem.querySelector('.status-text');
const statusIcon = fileItem.querySelector('.status-icon');
statusText.textContent = '计算MD5...';
const spark = new SparkMD5.ArrayBuffer();
const fileReader = new FileReader();
const chunkSize = 10 * 1024 * 1024; // MD5计算块大小
const totalChunks = Math.ceil(file.size / chunkSize);
let currentChunk = 0;
const processChunk = () => {
const start = currentChunk * chunkSize;
const end = Math.min(start + chunkSize, file.size);
fileReader.readAsArrayBuffer(file.slice(start, end));
};
fileReader.onload = (e) => {
spark.append(e.target.result);
currentChunk++;
const progress = Math.round((currentChunk / totalChunks) * 100);
statusText.textContent = `计算MD5: ${progress}%`;
if (currentChunk < totalChunks) {
processChunk();
} else {
const md5 = spark.end();
fileObj.md5 = md5;
statusText.textContent = 'MD5计算完成';
statusIcon.innerHTML = '';
// 准备分片
this.prepareChunks(file);
// 显示上传按钮
fileItem.querySelector('.upload-btn').classList.remove('hidden');
fileItem.querySelector('.cancel-btn').classList.remove('hidden');
}
};
fileReader.onerror = () => {
statusText.textContent = 'MD5计算失败';
statusIcon.innerHTML = '';
};
processChunk();
}
// 准备文件分片
prepareChunks(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const totalSize = file.size;
const totalChunks = Math.ceil(totalSize / this.chunkSize);
fileObj.chunks = [];
for (let i = 0; i < totalChunks; i++) {
const start = i * this.chunkSize;
const end = Math.min(start + this.chunkSize, totalSize);
fileObj.chunks.push({
index: i,
start,
end,
size: end - start,
status: 'pending',
retries: 0
});
}
fileObj.totalChunks = totalChunks;
}
// 开始上传文件
startUpload(file) {
const fileObj = this.getFileObject(file);
if (!fileObj || fileObj.status === 'uploading') return;
fileObj.status = 'uploading';
const fileItem = fileObj.element;
fileItem.querySelector('.status-text').textContent = '上传中...';
fileItem.querySelector('.status-icon').innerHTML = '';
fileItem.querySelector('.upload-btn').disabled = true;
// 并发上传分片
this.uploadChunksConcurrently(file);
}
// 并发上传分片
uploadChunksConcurrently(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const pendingChunks = fileObj.chunks.filter(chunk => chunk.status === 'pending');
const activeUploads = this.getActiveUploads(file);
// 如果所有分片都上传完成，发起合并请求
if (pendingChunks.length === 0 && activeUploads === 0) {
this.mergeChunks(file);
return;
}
// 控制并发数
while (activeUploads < this.concurrency && pendingChunks.length > 0) {
const chunk = pendingChunks[0];
this.uploadChunk(file, chunk);
// 标记为上传中
chunk.status = 'uploading';
}
}
// 获取活跃的上传数
getActiveUploads(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return 0;
return fileObj.chunks.filter(chunk => chunk.status === 'uploading').length;
}
// 上传单个分片
uploadChunk(file, chunk) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const formData = new FormData();
formData.append('file', file.slice(chunk.start, chunk.end), file.name);
formData.append('fileName', file.name);
formData.append('chunkNumber', chunk.index);
formData.append('totalChunks', fileObj.totalChunks);
formData.append('md5', fileObj.md5);
fetch(this.uploadUrl, {
method: 'POST',
body: formData
})
.then(response => {
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return response.json();
})
.then(data => {
if (data.success) {
// 分片上传成功
chunk.status = 'success';
// 更新进度
this.updateUploadProgress(file);
// 继续上传其他分片
this.uploadChunksConcurrently(file);
} else {
throw new Error(data.message || '上传失败');
}
})
.catch(error => {
console.error('上传分片失败:', error);
// 重试机制
if (chunk.retries < 3) {
chunk.retries++;
chunk.status = 'pending';
// 延迟重试
setTimeout(() => {
this.uploadChunk(file, chunk);
}, 1000 * chunk.retries);
} else {
// 重试次数用尽
chunk.status = 'failed';
fileObj.status = 'failed';
const fileItem = fileObj.element;
fileItem.querySelector('.status-text').textContent = '上传失败';
fileItem.querySelector('.status-icon').innerHTML = '';
fileItem.querySelector('.upload-btn').disabled = false;
}
});
}
// 更新上传进度
updateUploadProgress(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const completedChunks = fileObj.chunks.filter(chunk => chunk.status === 'success').length;
const progress = Math.round((completedChunks / fileObj.totalChunks) * 100);
fileObj.progress = progress;
const fileItem = fileObj.element;
fileItem.querySelector('.progress-value').style.width = `${progress}%`;
fileItem.querySelector('.status-text').textContent = `上传中: ${progress}%`;
}
// 合并分片
mergeChunks(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
const fileItem = fileObj.element;
fileItem.querySelector('.status-text').textContent = '合并分片中...';
const formData = new FormData();
formData.append('fileName', file.name);
formData.append('totalChunks', fileObj.totalChunks);
formData.append('md5', fileObj.md5);
// 合并API通常是另一个端点
const mergeUrl = this.uploadUrl.replace('/chunk', '/merge');
fetch(mergeUrl, {
method: 'POST',
body: formData
})
.then(response => {
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return response.json();
})
.then(data => {
if (data.success) {
fileObj.status = 'completed';
fileItem.querySelector('.status-text').textContent = '上传完成';
fileItem.querySelector('.status-icon').innerHTML = '';
fileItem.querySelector('.upload-btn').classList.add('hidden');
fileItem.querySelector('.cancel-btn').classList.add('hidden');
// 添加下载链接（如果服务器返回了文件URL）
if (data.fileUrl) {
const downloadLink = document.createElement('a');
downloadLink.href = data.fileUrl;
downloadLink.target = '_blank';
downloadLink.className = 'btn-secondary px-3 py-1 text-sm';
downloadLink.innerHTML = '下载';
fileItem.querySelector('div:last-child').appendChild(downloadLink);
}
} else {
throw new Error(data.message || '合并失败');
}
})
.catch(error => {
console.error('合并分片失败:', error);
fileObj.status = 'failed';
fileItem.querySelector('.status-text').textContent = '合并失败';
fileItem.querySelector('.status-icon').innerHTML = '';
fileItem.querySelector('.upload-btn').disabled = false;
});
}
// 取消上传
cancelUpload(file) {
const fileObj = this.getFileObject(file);
if (!fileObj) return;
fileObj.status = 'cancelled';
// 移除文件项
setTimeout(() => {
fileObj.element.classList.add('opacity-0', 'translate-y-4');
fileObj.element.style.transition = 'all 0.5s ease-out';
setTimeout(() => {
fileObj.element.remove();
this.files = this.files.filter(item => item.file !== file);
}, 500);
}, 100);
}
// 获取文件对象
getFileObject(file) {
return this.files.find(item => item.file === file);
}
}
// 初始化上传管理器
document.addEventListener('DOMContentLoaded', () => {
const uploadManager = new FileUploadManager();
});
script>
body>
html>