Python 实现POST上行压缩上传可用HTTP库汇总
先重申核心原理上行请求体压缩不是HTTP库自带功能本质是你自己把body gzip压缩带上Content-Encoding: gzip请求头将压缩二进制作为请求体发送。所有支持自定义请求body、自定义请求头的HTTP客户端都能实现区别只是各个库传参名称不一样requestsdatahttpxcontentaiohttpdataurllibdatapycurlPOSTFIELDS下面分「同步库」「异步库」给出完整示例统一复用gzip压缩函数。通用压缩工具函数所有示例共用importgzipfromioimportBytesIOdefgzip_compress(raw_bytes:bytes)-bytes:bufBytesIO()withgzip.GzipFile(fileobjbuf,modewb)asf:f.write(raw_bytes)returnbuf.getvalue()一、标准库urllib无需额外安装内置无需安装第三方包适合环境受限不能装requests的场景importjsonfromurllibimportrequest urlhttp://127.0.0.1:8000/api/uploadpayload{data:list(range(2000))}rawjson.dumps(payload).encode(utf-8)compressed_bodygzip_compress(raw)headers{Content-Type:application/json; charsetutf-8,Content-Encoding:gzip}reqrequest.Request(url,datacompressed_body,headersheaders,methodPOST)withrequest.urlopen(req)asresp:print(resp.read().decode(utf-8))缺点API简陋不支持连接池、超时易用性差生产一般不首选。二、httpx同步 异步当前主流推荐新一代HTTP客户端兼容requests风格支持http1.1/http2importhttpximportjson urlhttp://127.0.0.1:8000/api/uploadpayload{data:list(range(2000))}rawjson.dumps(payload).encode(utf-8)compressed_bodygzip_compress(raw)headers{Content-Type:application/json; charsetutf-8,Content-Encoding:gzip}# 同步withhttpx.Client(timeout10)asclient:resclient.post(url,headersheaders,contentcompressed_body)print(res.text)# 异步版本 import asyncio async def run(): async with httpx.AsyncClient(timeout10) as client: res await client.post(url, headersheaders, contentcompressed_body) print(res.text) asyncio.run(run()) ⚠️ httpx 传二进制body参数名是content不是data三、aiohttp异步首选高并发场景异步生态标准HTTP客户端爬虫、异步微服务大量使用importaiohttpimportasyncioimportjsonasyncdefmain():urlhttp://127.0.0.1:8000/api/uploadpayload{data:list(range(2000))}rawjson.dumps(payload).encode(utf-8)compressed_bodygzip_compress(raw)headers{Content-Type:application/json; charsetutf-8,Content-Encoding:gzip}asyncwithaiohttp.ClientSession()assession:asyncwithsession.post(url,datacompressed_body,headersheaders)asresp:print(awaitresp.text())asyncio.run(main())四、pycurl底层封装libcurl高性能基于C库libcurl吞吐性能极强适合超高并发场景缺点Windows安装麻烦、API晦涩importpycurlimportjsonfromioimportBytesIO urlhttp://127.0.0.1:8000/api/uploadpayload{data:list(range(2000))}rawjson.dumps(payload).encode(utf-8)compressed_bodygzip_compress(raw)resp_bufferBytesIO()cpycurl.Curl()c.setopt(c.URL,url)c.setopt(c.POST,1)c.setopt(c.POSTFIELDS,compressed_body)c.setopt(c.WRITEDATA,resp_buffer)headers[Content-Type: application/json; charsetutf-8,Content-Encoding: gzip]c.setopt(c.HTTPHEADER,headers)c.perform()c.close()print(resp_buffer.getvalue().decode())五、tornado.httpclientTornado框架内置异步客户端如果你项目本身使用Tornado Web框架可以直接使用内置客户端无需额外依赖importjsonfromtornado.httpclientimportAsyncHTTPClient,HTTPRequestimporttornado.ioloopasyncdeffetch():urlhttp://127.0.0.1:8000/api/uploadpayload{data:list(range(2000))}rawjson.dumps(payload).encode(utf-8)compressed_bodygzip_compress(raw)reqHTTPRequest(url,methodPOST,bodycompressed_body,headers{Content-Type:application/json; charsetutf-8,Content-Encoding:gzip})clientAsyncHTTPClient()resawaitclient.fetch(req)print(res.body.decode())tornado.ioloop.IOLoop.current().run_sync(fetch)关键共性注意事项所有库通用不要使用框架自带json参数例如httpx.post(json{})、requests.post(json{})这类接口会自动编码无法替换成压缩二进制必须手动序列化使用原始body参数。下行压缩 ≠ 上行压缩Accept-Encoding只是告诉服务端我能接收压缩响应Content-Encoding: gzip才代表本次请求body已经压缩。服务端必须手动解压绝大多数Web框架默认不会自动解压带有Content-Encoding的请求体缺少解压代码会直接收到乱码二进制。库选型建议新项目、同步异步都需要 →httpx纯异步高并发服务/爬虫 →aiohttp不能安装第三方包、极简环境 →urllib标准库极致性能、熟悉libcurl →pycurl老旧项目维持现状 → requests
