ThinkPHP导出百万条数据量
2021-04-05 20:26
标签:权限 func fclose enc 技术 limit for 溢出 tps 项目中,需要将数据库中的百万条数据导出,用PHPExcel的话,PHP内存会各种溢出什么的,所以就用的CSV批量导出再打包下载,随后删除生成的数据。 原文地址:https://blog.csdn.net/lzj_dandan/article/details/78490687 ThinkPHP导出百万条数据量 标签:权限 func fclose enc 技术 limit for 溢出 tps 原文地址:https://www.cnblogs.com/panziwen/p/12505226.html
public function excelout()
{
$array = array(‘ulId‘, ‘aucSUPI‘, ‘ucRatRest‘, ‘ausForbidden‘, ‘ausForbidde‘, ‘ausAllowedArea_1‘,
‘ausAllowedArea_2‘, ‘ausNonAl‘, ‘ausNonAllow‘, ‘ausAllowN‘, ‘ausAllow‘, ‘ucMicoMode‘,‘aucDnn_1‘, ‘aucDnn_2‘, ‘aucDnn_3‘, ‘aucLand‘,‘ulTa1‘,‘ulTa2‘
);
$this->putCsv($array);
}
function putCsv($head, $mark = ‘attack_ip_info‘, $fileName = "test.csv")
{
set_time_limit(0);
$sqlCount = Db::table(‘ue_subscriber‘)->count();
//输出Excel文件头,可把user.csv换成你要的文件名
header(‘Content-Type: application/vnd.ms-excel;charset=utf-8‘);
header(‘Content-Disposition: attachment;filename="‘ . $fileName . ‘"‘);
header(‘Cache-Control: max-age=0‘);
$sqlLimit = 100000;//每次只从数据库取100000条以防变量缓存太大
//每隔$limit行,刷新一下输出buffer,不要太大,也不要太小
$limit = 100000;
//buffer计数器
$cnt = 0;
$fileNameArr = array();
//逐行取出数据,不浪费内存
for ($i = 0; $i ceil($sqlCount / $sqlLimit); $i++) {
$fp = fopen($mark . ‘_‘ . $i . ‘.csv‘, ‘w‘); //生成临时文件
//chmod(‘attack_ip_info_‘ . $i . ‘.csv‘,777);//修改可执行权限
$fileNameArr[] = $mark . ‘_‘ . $i . ‘.csv‘;
//将数据通过fputcsv写到文件句柄
fputcsv($fp, $head);
$dataArr = Db::table(‘ue_subscriber‘)->limit($i * $sqlLimit,$sqlLimit)->select();
foreach ($dataArr as $a) {
$cnt++;
if ($limit == $cnt) {
//刷新一下输出buffer,防止由于数据过多造成问题
ob_flush();
flush();
$cnt = 0;
}
fputcsv($fp, $a);
}
fclose($fp);//每生成一个文件关闭
}
//进行多个文件压缩
$zip = new ZipArchive();
$filename = $mark . ".zip";
$zip->open($filename, ZipArchive::CREATE);//打开压缩包
foreach ($fileNameArr as $file) {
$zip->addFile($file, basename($file));//向压缩包中添加文件
}
$zip->close();//关闭压缩包
foreach ($fileNameArr as $file) {
unlink($file);//删除csv临时文件
}
//输出压缩文件提供下载
header("Cache-Control: max-age=0");
header("Content-Description: File Transfer");
header(‘Content-disposition: attachment; filename=‘ . basename($filename));
header("Content-Type: application/zip");
header("Content-Transfer-Encoding: binary");
header(‘Content-Length: ‘ . filesize($filename));
@readfile($filename);//输出文件;
unlink($filename); //删除压缩包临时文件
}
下一篇:C#获取枚举描述