How do I continuously pipe to curl and post with chunking?
Asked Answered
L

0

7

I know that curl can post data to a URL:

$ curl -X POST http://httpbin.org/post -d "hello"
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "hello": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "5", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.50.1"
  }, 
  "json": null, 
  "origin": "64.238.132.14", 
  "url": "http://httpbin.org/post"
}

And I know I can pipe to curl to achieve the same thing:

$ echo "hello" | curl -X POST http://httpbin.org/post -d @-
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "hello": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "5", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.50.1"
  }, 
  "json": null, 
  "origin": "64.238.132.14", 
  "url": "http://httpbin.org/post"
}

Now here is where it gets tricky. I know about http transfer-coding and chunking, for example to send a multiline file:

('silly' is a file that contains several lines of text, as seen here)

$ cat silly | curl -X POST --header "Transfer-Encoding: chunked" "http://httpbin.org/post" --data-binary @-
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "hello\nthere\nthis is a multiple line\nfile\n": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "41", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.50.1"
  }, 
  "json": null, 
  "origin": "64.238.132.14", 
  "url": "http://httpbin.org/post"
}

Now what I want to be able to do is to have curl read a line from stdin, send it as a chunk, and then come back and read stdin again (which allows me to keep it going continously). This was my first attempt:

curl -X POST --header "Transfer-Encoding: chunked" "http://httpbin.org/post" -d @-

And it does work ONE TIME ONLY when I hit ctrl-D, but that obviously ends the execution of curl.

Is there any way to tell curl "Send (using chunk encoding) what I've given you so far, and then come back to stdin for more" ?

Thanks so much, I've been scratching my head for a while on this one!

Laminitis answered 9/12, 2016 at 21:48 Comment(3)
you can do it with proc_open() & coIsopropyl
Do you have a list of files that you want to send all at once in separate chunks, or do you want to keep a stream open to write to at any point in the future?Burden
I want to keep a stream open and write to it at any point in the future.Laminitis

© 2022 - 2024 — McMap. All rights reserved.