How to Pipe Data to cURL POST Requests

1. Overview

cURL is a highly versatile command-line tool for interacting with HTTP APIs. A particularly useful yet sometimes overlooked feature is its ability to pipe data to POST requests directly. This functionality proves invaluable for dynamic scripting, handling streaming inputs, and creating command pipelines within shell environments.

In this tutorial, we’ll use the REST APIs from httpbin.org to learn how to pipe data while making the POST requests with the curl command.

2. Using the –data Option

Before we start, we must note that the responses from httpbin.org include the request parameters, which will help us understand how the server interpreted our request.

We can use the @- parameter with the –data option to receive data sent across the pipe:

$ echo '{"name":"Alice"}' | curl -X POST --data @- https://httpbin.org/anything

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "{\"name\":\"Alice\"}": ""
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "16",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-681560d9-21e75bb25fde1a3f7695b99b"
  },
  "json": null,
  "method": "POST",
  "origin": "223.190.87.138",
  "url": "https://httpbin.org/anything"
}

We used the echo command to send a JSON object. Even though the content was a JSON object, the Content-Type request header was sent as application/x-www-form-urlencoded because of the default behavior of the —data option. As a result, the server interpreted the content as form data.

Next, let’s use the –header option to overwrite the Content-Type request header:

$ echo '{"name":"Alice"}' | curl -X POST --header "Content-Type: application/json" --data @- https://httpbin.org/anything

{
  "args": {},
  "data": "{\"name\":\"Alice\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "16",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-68156151-366375de3220afc4609d7493"
  },
  "json": {
    "name": "Alice"
  },
  "method": "POST",
  "origin": "223.190.87.138",
  "url": "https://httpbin.org/anything"
}

Great! The server understood the request content correctly, and we can see that the interpreted data on the server side is a JSON string.

3. Using the –data-binary Option

Although the –data option is the most common option for POST data, it removes the newlines from the input. As a result, it’s not suitable for use cases where we want to preserve the input as it is.

Let’s understand the problem with the help of the multi-line input available in the sample.txt file:

$ cat sample.txt
line-1
line-2
line-3
line-4

Although we could also send multi-line input over the command line, sending it via a file is usually preferred so we don’t pollute the command history.

Now, let’s first use the –data option to send the content from the sample.txt file:

$ cat sample.txt | curl -X POST --header "Content-Type: text/plain" --data @- https://httpbin.org/anything
{
  "args": {},
  "data": "line-1line-2line-3line-4",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "24",
    "Content-Type": "text/plain",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-68156d94-0605072b1fb23fc57b77c2b9"
  },
  "json": null,
  "method": "POST",
  "origin": "223.190.87.138",
  "url": "https://httpbin.org/anything"
}

We can now acknowledge the issue caused by the –data option, as the response shows that the server received the content as a single line.

Lastly, let’s resolve this issue by using the –data-binary option, as it preserves the input exactly as-is:

$ cat sample.txt | curl -X POST --header "Content-Type: text/plain" --data-binary @- https://httpbin.org/anything
{
  "args": {},
  "data": "line-1\nline-2\nline-3\nline-4\n",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "28",
    "Content-Type": "text/plain",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-68156db2-226e5489529177872388299c"
  },
  "json": null,
  "method": "POST",
  "origin": "223.190.87.138",
  "url": "https://httpbin.org/anything"
}

Perfect! It looks like we nailed this one. The response shows that the server received a multi-line string in the content.

4. Using the –json Option

Since JSON is the most common format for working with HTTP APIs, the cURL 7.82.0 version introduced the –json option to facilitate working with JSON data.

Let’s look at the test content in the sample.json file:

$ cat sample.json
{
  "key": "value"
}

We have a tab indentation to see how it’d be interpreted when sending this content to the server.

Now, let’s use the –json option along with @- to pipe the data from the sample.json file to curl, making the POST request:

$ cat sample.json | curl -X POST --json @- https://httpbin.org/anything
{
  "args": {},
  "data": "{\n\t\"key\": \"value\"\n}\n",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json",
    "Content-Length": "20",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/8.7.1",
    "X-Amzn-Trace-Id": "Root=1-68157371-5a749ac708a515de51bfcb0d"
  },
  "json": {
    "key": "value"
  },
  "method": "POST",
  "origin": "223.190.87.138",
  "url": "https://httpbin.org/anything"
}

The server interpreted the JSON content correctly. Moreover, it preserved both tabs and newlines in the input as-is.

5. Conclusion

In this tutorial, we learned how to pipe data to cURL POST requests. Further, we explored some of the popular options, such as –data, —data-binary, and —json, while solving our use case.

Firstly, we can use –data-binary for most of our use cases while explicitly setting the Content-Type request header. Nonetheless, if our use case can afford to miss newlines in the input, it’s better to use –data as it could reduce the request size for large payloads. Lastly, if we’re working with JSON data, using –json is preferred as it sets the Content-Type header automatically.