从文件加密到到视频文件进度条播放揭秘
文件加密
使用 Cipher CipherInputStream CipherOutputStream 实现对文件的加解密
每个文件使用一个秘钥 String aesKey = UUID.randomUUID().toString().replace("-",""); 可以通过uuid or 其他的途径生成一个唯一的秘钥。
- 文件的加解密
private static final String ALGORITHM_STREAM = "AES/ECB/PKCS5Padding";
/**
* 加密数据
*
* @param input
* @param key
* @return
* @throws Exception
*/
public static InputStream encodeStream(InputStream input, String key) throws Exception {
SecretKey secretKey = generateAesKey(key);
Cipher c = Cipher.getInstance(ALGORITHM_STREAM);
c.init(1, secretKey);
return new CipherInputStream(input, c);
}
/**
* 解密文件流信息
*
* @param input
* @param key
* @return
* @throws Exception
*/
public static InputStream decodeStream(InputStream input, String key) throws Exception {
SecretKey secretKey = generateAesKey(key);
Cipher c = Cipher.getInstance(ALGORITHM_STREAM);
c.init(2, secretKey);
return new CipherInputStream(input, c);
}
private static SecretKey generateAesKey(String key) {
try {
KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
SecureRandom secureRandom = SecureRandom.getInstance("SHA1PRNG");
secureRandom.setSeed(key.getBytes());
keyGenerator.init(128, secureRandom);
return keyGenerator.generateKey();
} catch (GeneralSecurityException e) {
throw new RuntimeException(e);
}
}
视频分段渐进式播放
样例eg： http://mirror.aarnet.edu.au/pub/TED-talks/911Mothers_2010W-480p.mp4
当前这个视频播放实现随机播放、实现分块下载等等能力，一般情况下后端下载视频
http://localhost:8080/web/file-upload/common-file-download?fileId=ab87ef175dc1419b922acb35dd3ad58e
提供类似的URL地址 后端直接写流到浏览器 IOUtils.copy(encodeInputStream, response.getOutputStream());
当点击视频中进度条的时候永远都不行，点击进度条相当于重新请求、视频流信息重0开始，直接破坏了进度条的规则。
查询了一些资料，其实视频播放的时候通过Range 进行了分段的请求、我们可以对于流量进行分段的下载处理
- Java后端实现视频分段渐进式播放
- HTTP 方式文件分片断点下载附JAVA实现
- java http Range分段下载
206 状态码 & Content-Range 分段的响应下载,读取当前流信息中指定开始到指定长度的流信息
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray());
@GetMapping(value = "/common-file-download")
@ResponseBody
public void commonFileDownload(@RequestParam String fileId, HttpServletRequest request, HttpServletResponse response) throws Exception {
File desc = new File(System.getProperty("user.dir") + "/file/" + fileId);
//获取从那个字节开始读取文件
try (FileInputStream inputStream = new FileInputStream(desc)) {
InputStream encodeInputStream = AesUtils.decodeStream(inputStream, fileId);
int fSize = FILE_LENGTH;
if (this.haveRanges(request)) {
// 断点续传
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), fSize, 5 * 1024 * 1024);
//开始下载位置
int startByte = rangeInfo.getRight();
//结束下载位置
int endByte = rangeInfo.getLeft();
//要下载的长度
int contentLength = endByte - startByte + 1;
//Content-Length 表示资源内容长度，即：文件大小
response.setHeader(HttpHeaders.CONTENT_LENGTH, contentLength + "");
//Content-Range 表示响应了多少数据，格式为：[要下载的开始位置]-[结束位置]/[文件总大小]
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + startByte + "-" + endByte + "/" + fSize);
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "inline;filename=" + fileName);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray());
return;
}
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copy(encodeInputStream, response.getOutputStream());
} catch (ClientAbortException e) {
log.debug("ignore {}", e.getMessage());
} catch (Exception e) {
log.error("error", e);
} finally {
IOUtils.close(response.getOutputStream());
}
}
/**
* 获取 range 的长度信息
*
* @param range
* @param fileSize
* @param defaultRangeLengthSize
* @return
*/
public MutablePair<Integer, Integer> getRangeInfo(String range, int fileSize, int defaultRangeLengthSize) {
MutablePair<Integer, Integer> rangeInfo = new MutablePair<>();
rangeInfo.setLeft(fileSize - 1);
rangeInfo.setRight(0);
if (StringUtils.isNotBlank(range) && range.contains("bytes=") && range.contains("-")) {
range = range.substring(range.lastIndexOf("=") + 1).trim();
String[] ranges = range.split("-");
int startByte = 0;
int endByte = fileSize - 1;
try {
//根据range解析下载分片的位置区间
if (ranges.length == 1) {
//情况1，如：bytes=-1024 从开始字节到第1024个字节的数据
if (range.startsWith("-")) {
endByte = Integer.parseInt(ranges[0]);
}
//情况2，如：bytes=1024- 第1024个字节到最后字节的数据
else if (range.endsWith("-")) {
startByte = Integer.parseInt(ranges[0]);
//增加一个默认的信息
endByte = startByte + defaultRangeLengthSize;
if (endByte >= fileSize - 1) {
endByte = fileSize - 1;
}
}
}
//情况3，如：bytes=1024-2048 第1024个字节到2048个字节的数据
else if (ranges.length == 2) {
startByte = Integer.parseInt(ranges[0]);
endByte = Integer.parseInt(ranges[1]);
}
} catch (NumberFormatException e) {
startByte = 0;
endByte = fileSize - 1;
}
rangeInfo.setRight(startByte);
rangeInfo.setLeft(endByte);
}
return rangeInfo;
}
/**
* 是否有Range
*
* @param request
* @return
*/
public boolean haveRanges(HttpServletRequest request) {
String range = request.getHeader(HttpHeaders.RANGE);
if (StringUtils.isNotBlank(range) && range.contains("bytes=") && range.contains("-")) {
return true;
}
return false;
}
感觉一切都很好了… 本地文件也可以了…
块存储文件服务器加密文件分块下下载
第一个版本 【进度条点击越靠后的时候 响应的时间越来越长】
下载非常的慢
http://localhost:8080/web/file-upload/version1-file-download
为什么?主要是下面这个代码每次都需要下载跳过的流文件进行解密,随着长度的加深 速度越来越慢
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray())
/**
* 下载比较慢
*
* @param fileId
* @param request
* @param response
* @throws Exception
*/
@GetMapping(value = "/version1-file-download")
@ResponseBody
public void version1(@RequestParam(required = false) String fileId, HttpServletRequest request, HttpServletResponse response) throws Exception {
File desc = new File(System.getProperty("user.dir") + "/file/" + fileId);
fileId = "Qq87r0SidfdM9i5QDrtKLTbRpcGam0qd";
S3Object object = s3.getObject(BUCK_NAME, fileId);
try (InputStream inputStream = object.getObjectContent()) {
InputStream encodeInputStream = AesUtils.decodeStream(inputStream, fileId);
int fSize = FILE_LENGTH;
if (this.haveRanges(request)) {
// 断点续传
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), fSize, 5 * 1024 * 1024);
//开始下载位置
int startByte = rangeInfo.getRight();
//结束下载位置
int endByte = rangeInfo.getLeft();
//要下载的长度
int contentLength = endByte - startByte + 1;
//Content-Length 表示资源内容长度，即：文件大小
response.setHeader(HttpHeaders.CONTENT_LENGTH, contentLength + "");
//Content-Range 表示响应了多少数据，格式为：[要下载的开始位置]-[结束位置]/[文件总大小]
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + startByte + "-" + endByte + "/" + fSize);
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "inline;filename=" + fileName);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray());
return;
}
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copy(encodeInputStream, response.getOutputStream());
} catch (ClientAbortException e) {
log.debug("ignore {}", e.getMessage());
} catch (Exception e) {
log.error("error", e);
} finally {
IOUtils.close(response.getOutputStream());
}
}
第二个版本 偶尔出行错误 & aes 解密 没有16块 & 播放偶现突然停止
http://localhost:8080/web/file-upload/version2-file-download
为了解决下载响应非常慢的问题，需要通过s3 支持的分块下载
支持通过header 进行分块下载…
GetObjectRequest getObjectRequest = new GetObjectRequest(BUCK_NAME, fileId);
int startOffset = rangeInfo.getRight();
getObjectRequest.setRange(startOffset);
object = s3.getObject(getObjectRequest);
感觉好像好了，其实还有问题，任意拖住进度条一会就不能播放了…自然就停止了…
看到报错,不过这个问题很关键 下载的内容长度没有16的整数倍
java.io.IOException: javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
at java.base/javax.crypto.CipherInputStream.getMoreData(CipherInputStream.java:128)
at java.base/javax.crypto.CipherInputStream.read(CipherInputStream.java:242)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1384)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1342)
**
* 下载偶尔失败
*
* @param fileId
* @param request
* @param response
* @throws Exception
*/
@GetMapping(value = "/version2-file-download")
@ResponseBody
public void version2(@RequestParam(required = false) String fileId, HttpServletRequest request, HttpServletResponse response) throws Exception {
File desc = new File(System.getProperty("user.dir") + "/file/" + fileId);
fileId = "Qq87r0SidfdM9i5QDrtKLTbRpcGam0qd";
S3Object object = null;
boolean isRange3s = false;
if (haveRanges(request)) {
GetObjectRequest getObjectRequest = new GetObjectRequest(BUCK_NAME, fileId);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), FILE_LENGTH, 5 * 1024 * 1024);
int startOffset = rangeInfo.getRight();
getObjectRequest.setRange(startOffset);
object = s3.getObject(getObjectRequest);
isRange3s = true;
} else {
object = s3.getObject(BUCK_NAME, fileId);
}
try (InputStream inputStream = object.getObjectContent()) {
InputStream encodeInputStream = AesUtils.decodeStream(inputStream, fileId);
int fSize = FILE_LENGTH;
if (this.haveRanges(request)) {
// 断点续传
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), fSize, 5 * 1024 * 1024);
//开始下载位置
int startByte = rangeInfo.getRight();
//结束下载位置
int endByte = rangeInfo.getLeft();
//要下载的长度
int contentLength = endByte - startByte + 1;
//Content-Length 表示资源内容长度，即：文件大小
response.setHeader(HttpHeaders.CONTENT_LENGTH, contentLength + "");
//Content-Range 表示响应了多少数据，格式为：[要下载的开始位置]-[结束位置]/[文件总大小]
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + startByte + "-" + endByte + "/" + fSize);
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "inline;filename=" + fileName);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
if (isRange3s) {
// Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
// 忽略这个错误..
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), 0, contentLength, IOUtils.byteArray());
} else {
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray());
}
return;
}
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copy(encodeInputStream, response.getOutputStream());
} catch (ClientAbortException e) {
log.debug("ignore {}", e.getMessage());
} catch (Exception e) {
log.error("error", e);
} finally {
IOUtils.close(response.getOutputStream());
}
}
最后一个版本，解决问题
为什么第二个版本有问题？
AES wikipedia
AES加密过程是在一个4×4的字节矩阵上运作，比如【1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16】16个字节一起加密，没有16个字节进行填充处理。
AES 加密后的大小=(AES 加密前的大小/16+1)*16 按照这样的说法、解密的时候只能 1~16一起解密，不能 2~17一起解密。因此要计算当前 startRange 最小的16块的起始位置作为开始点
进行Range 下载 startOffset =(startByte / 16) * 16 ,这样下载后其实文件流多读取了一部分，所以在响应的时候要跳过这个部分
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte - startOffset, contentLength, IOUtils.byteArray()); 整体上就完美的解密、完美定位到具体的字节流的信息了。
http://localhost:8080/web/file-upload/version3-file-download
/**
* 正常版本
*
* @param fileId
* @param request
* @param response
* @throws Exception
*/
@GetMapping(value = "/version3-file-download")
@ResponseBody
public void version3(@RequestParam(required = false) String fileId, HttpServletRequest request, HttpServletResponse response) throws Exception {
File desc = new File(System.getProperty("user.dir") + "/file/" + fileId);
fileId = "Qq87r0SidfdM9i5QDrtKLTbRpcGam0qd";
S3Object object = null;
boolean isRange3s = false;
if (haveRanges(request)) {
GetObjectRequest getObjectRequest = new GetObjectRequest(BUCK_NAME, fileId);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), FILE_LENGTH, 5 * 1024 * 1024);
int startOffset = rangeInfo.getRight();
startOffset = (startOffset / 16) * 16;
getObjectRequest.setRange(startOffset);
object = s3.getObject(getObjectRequest);
isRange3s = true;
} else {
object = s3.getObject("owork-file-demo", fileId);
}
try (InputStream inputStream = object.getObjectContent()) {
InputStream encodeInputStream = AesUtils.decodeStream(inputStream, fileId);
int fSize = FILE_LENGTH;
if (this.haveRanges(request)) {
// 断点续传
response.setStatus(HttpServletResponse.SC_PARTIAL_CONTENT);
MutablePair<Integer, Integer> rangeInfo = this.getRangeInfo(request.getHeader(HttpHeaders.RANGE), fSize, 5 * 1024 * 1024);
//开始下载位置
int startByte = rangeInfo.getRight();
//结束下载位置
int endByte = rangeInfo.getLeft();
//要下载的长度
int contentLength = endByte - startByte + 1;
int startOffset = (startByte / 16) * 16;
//Content-Length 表示资源内容长度，即：文件大小
response.setHeader(HttpHeaders.CONTENT_LENGTH, contentLength + "");
//Content-Range 表示响应了多少数据，格式为：[要下载的开始位置]-[结束位置]/[文件总大小]
response.setHeader(HttpHeaders.CONTENT_RANGE, "bytes " + startByte + "-" + endByte + "/" + fSize);
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
response.setHeader(HttpHeaders.CONTENT_DISPOSITION, "inline;filename=" + fileName);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
if (isRange3s) {
// Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
// 忽略这个错误..
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte - startOffset, contentLength, IOUtils.byteArray());
} else {
IOUtils.copyLarge(encodeInputStream, response.getOutputStream(), startByte, contentLength, IOUtils.byteArray());
}
return;
}
String fileName = URLEncoder.encode("test.mp4", StandardCharsets.UTF_8);
String mimetype = Mimetypes.getInstance().getMimetype(fileName);
response.setContentType(mimetype);
IOUtils.copy(encodeInputStream, response.getOutputStream());
} catch (ClientAbortException e) {
log.debug("ignore {}", e.getMessage());
} catch (Exception e) {
log.error("error", e);
} finally {
IOUtils.close(response.getOutputStream());
}
}
总结
从文件加密到视频分段视频分段渐进式播放问题一路探究,了解了文件的分片下载的实现原理，对于加密文件的处理如果实现分片下载失败原理进行探究。
参考文档
- 文件的加解密
- 视频demo格式下载
- Java后端实现视频分段渐进式播放
- HTTP 方式文件分片断点下载附JAVA实现
- java http Range分段下载