HTTPie and Print HTTP Request

   |  1521 words  |  Kracekumar

HTTPie is a command-line utility for making HTTP requests with more straightforward syntax(controversial, I agree). The interesting feature is --offline flag which prints HTTP raw request text. The client sends the HTTP request to the server, and the server responds to the request. It’s an alternate to curl.

HTTP Syntax

HTTP Flow and syntrax from Wikipedia.

A client sends request messages to the server, which consist of

  • a request line, consisting of the case-sensitive request method, a space, the request target, another space, the protocol version, a carriage return, and a line feed (e.g. GET /images/logo.png HTTP/1.1)
  • zero or more request header fields, each consisting of the case-insensitive field name, a colon, optional leading whitespace, the field value, and optional trailing whitespace (e.g. Accept-Language: en), and ending with a carriage return and a line feed.
  • an empty line, consisting of a carriage return and a line feed;
  • an optional message body.
  • In the HTTP/1.1 protocol, all header fields except Host are optional.
  • A request line containing only the path name is accepted by servers to maintain compatibility with HTTP clients before the HTTP/1.0 specification in RFC 1945.

Throughout the post, I’ll use --offline feature to understand how the HTTP request structure looks for educational purposes.

🉑 Accept only JSON response data 🉑

$http --offline httpbin.org/get Accept:application/json
GET /get HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/2.5.0
# Sample request sent to the server
$ http httpbin.org/get Accept:application/json
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 310
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:21:40 GMT
Server: gunicorn/19.9.0

{
    "args": {},
    "headers": {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/2.5.0",
        "X-Amzn-Trace-Id": "Root=1-61629484-3b25a3631e2a89bf60f2600e"
    },
    "origin": "xxx.xxx.xxx.xxx",
    "url": "http://httpbin.org/get"
}

  • You can pass more than one value in the Accept header seperated by comma.

  • The response from the server contains Content-Type as JSON. The server can choose to ignore the Accept header.

📚 Request the Tamil language version of duckduckgo 📚

$http --offline  https://duckduckgo.com Accept-Language:ta
GET / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: ta
Connection: keep-alive
Host: duckduckgo.com
User-Agent: HTTPie/2.5.0

$http  https://duckduckgo.com Accept-Language:ta
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Encoding: gzip
Content-Security-Policy: default-src 'none' ; connect-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; manifest-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; media-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; script-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' 'unsafe-eval' ; font-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; img-src data:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; style-src  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; frame-src blob:  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ ; form-action  https://duckduckgo.com https://*.duckduckgo.com https://3g2upl4pq6kufc4m.onion/ https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com/ https://duck.co ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;
Content-Type: text/html; charset=UTF-8
Date: Sat, 09 Oct 2021 21:44:33 GMT
ETag: W/"6161a338-16b8"
Expect-CT: max-age=0
Expires: Sat, 09 Oct 2021 21:44:32 GMT
Permissions-Policy: interest-cohort=()
Referrer-Policy: origin
Server: nginx
Strict-Transport-Security: max-age=31536000
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1;mode=block

<!DOCTYPE html>
...
<body id="pg-index" class="page-index body--home">
	<script type="text/javascript" src="/tl5.js"></script>
<script type="text/javascript" src="/lib/l123.js"></script>
<script type="text/javascript" src="/locale/ta_IN/duckduckgo50.js"></script>
<script type="text/javascript" src="/util/u588.js"></script>
<script type="text/javascript" src="/d3012.js"></script>
...
<noscript>
    <div class="tag-home">
        <div class="tag-home__wrapper">
            <div class="tag-home__item">
                Privacy, simplified&period;
                <span class="hide--screen-xs"><a href="/about" class="tag-home__link"> மேலும் கற்க</a></span>
            </div>
        </div>
    </div>
</noscript>
...
</html>
  • The header can carry more than one value separated by a comma.
  • When more than one value is present, an extra parameter q=0.5 represents the weightage among the values. Example: Accept-Language: ta,fr;q=0.50.
  • One of the link description is in Tamil, மேலும் கற்க.

🔑 Post Authorization token as part of login 🔑

$http --offline POST httpbin.org/auth Authorization:"Bearer my-dear-darkness"
POST /auth HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Authorization: Bearer my-dear-darkness
Connection: keep-alive
Content-Length: 0
Host: httpbin.org
User-Agent: HTTPie/2.5.0

🍪 Send Cookie as part of a request 🍪

$http --offline http://pie.dev/cookies Cookie:"name=not-unique;value=23"
GET /cookies HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: name=not-unique;value=23
Host: dev.pie
User-Agent: HTTPie/2.5.0
$http http://pie.dev/cookies Cookie:"name=not-unique;value=23"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1b984c743c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:24:13 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=2Zt9wQcKQYStY5dPukoEXFwn7K6pKNaVTiKljZ5h9oL3mcQZj0khYq8Kmzo8PmQinPncStZeHASetQqHzCODe0wbrljPEIJxCWGdRWMbry9rWOG%2FBheDvJs7"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "cookies": {
        "name": "not-unique",
        "value": "23"
    }
}

✉️ Send JSON Request ✉️

  • HTTPie supports sending form-encoded values or JSON values. = sign indiciates JSON key-value pair.
  • = is useful for primitive values like number, string, null, boolean.
  • Example: lang-py
$http --offline PUT http://pie.dev/put lang=py version=3.10
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 33
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "version": "3.10"
}
$HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be1e017fb53c12-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:25:52 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GMW5PQHMwjxpVUeDG7uWo6M%2FiKLMzmQLd1e5BIG3AXljmuQgwCP9nzrFPdaidR2wL14eisfiViDhumDHpepgNB6yIrsVKXRybHa5tRqmH7lUDoQdRqCK1Ijg"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\"}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be1e017fb53c12-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "33",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "version": "3.10"
    },
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/put"
}

✉️ Send non-primitive JSON values ✉️

  • := means the JSON value is non-primitve values like array/list and dictionary.
  • Single quote carries the value.
  • Example: os:='["GNU/Linux", "Mac OSX"]'.
$http --offline  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
PUT /put HTTP/1.1
Accept: application/json, */*;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 65
Content-Type: application/json
Host: pie.dev
User-Agent: HTTPie/2.5.0

{
    "lang": "py",
    "os": [
        "GNU/Linux",
        "Mac OSX"
    ],
    "version": "3.10"
}
$http  PUT http://pie.dev/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]'
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be224ce97c3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:28:48 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=dgY0R%2FqomBbumKs0IVB7GjPR2chhHJ9wVKAzW33IOnvZ%2Fs4l2LYnvKKeXo6Xhd162AYKGzyIrK4pNrYdH8SEs1NvGmYfJqEDIPmfUOELqpC6HK9iP1zIENa0"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
    "files": {},
    "form": {},
    "headers": {
        "Accept": "application/json, */*;q=0.5",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be224ce97c3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "65",
        "Content-Type": "application/json",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": {
        "lang": "py",
        "os": [
            "GNU/Linux",
            "Mac OSX"
        ],
        "version": "3.10"
    },
    "origin": "xx.xxx.xxx.xxx",
    "url": "http://pie.dev/put"
}

📤 Upload files 📤

  • -f flag indicates the data is form-encoded values.
  • @ symbol indicates the value is a file.
  • Example: cv@hello-world.txt. cv form field name.
$cat hello-world.txt
hello-world
$http --offline -f POST pie.dev/post name='Krace' cv@hello-world.txt
POST /post HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Content-Length: 277
Content-Type: multipart/form-data; boundary=183f70d3da41432d95bcd839e2cc20e2
Host: pie.dev
User-Agent: HTTPie/2.5.0

--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="name"

Krace
--183f70d3da41432d95bcd839e2cc20e2
Content-Disposition: form-data; name="cv"; filename="hello-world.txt"
Content-Type: text/plain

hello-world

--183f70d3da41432d95bcd839e2cc20e2--
$ http -f POST pie.dev/post name='Krace' cv@hello-world.txt
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 69be2564a8fc3c0c-BLR
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 10 Oct 2021 07:30:54 GMT
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=yNaTSZJ7ouYH%2FSJZw6LIR5vNl6KNTFeDKF1u8V60abb3ClKLzdOj0zkchcIAWqTvZDbxXm5MnffDkCLdqviMQuAo7DqFA2GH%2Bm%2FiZ6sH90oOr0HyFGAuS4Gp"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked
access-control-allow-credentials: true
access-control-allow-origin: *
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "args": {},
    "data": "",
    "files": {
        "cv": "hello-world\n"
    },
    "form": {
        "name": "Krace"
    },
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip",
        "Cdn-Loop": "cloudflare",
        "Cf-Connecting-Ip": "49.207.222.139",
        "Cf-Ipcountry": "IN",
        "Cf-Ray": "69be2564a8fc3c0c-FRA",
        "Cf-Visitor": "{\"scheme\":\"http\"}",
        "Connection": "Keep-Alive",
        "Content-Length": "277",
        "Content-Type": "multipart/form-data; boundary=252c6fcd1dcc40e09de54958660d672d",
        "Host": "pie.dev",
        "User-Agent": "HTTPie/2.5.0"
    },
    "json": null,
    "origin": "xxx.xxx.xxx.xxxx",
    "url": "http://pie.dev/post"
}

Explore more in HTTPie website.

🐈 How to verify the the generated request works? 🐈

Netcat is a utility for sending and receiving data in network connection using TCP or UDP. Netcat take hostname, port, and body as arguments and sends it to the server and displays the response.

$http --offline  PUT http://httpbin.org/put lang=py version=3.10 os:='["GNU/Linux", "Mac OSX"]' | nc httpbin.org 80
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2021 08:27:58 GMT
Content-Type: application/json
Content-Length: 631
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {},
  "data": "{\"lang\": \"py\", \"version\": \"3.10\", \"os\": [\"GNU/Linux\", \"Mac OSX\"]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "application/json, */*;q=0.5",
    "Accept-Encoding": "gzip, deflate",
    "Content-Length": "65",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "HTTPie/2.5.0",
    "X-Amzn-Trace-Id": "Root=1-6162a40e-34b9a83f40868b4a73e8fa09"
  },
  "json": {
    "lang": "py",
    "os": [
      "GNU/Linux",
      "Mac OSX"
    ],
    "version": "3.10"
  },
  "origin": "xxx.xxx.xxx.xxx",
  "url": "http://httpbin.org/put"
}

Netcat doesn’t support encrypted network connections.

References

Python  HTTP  CLI  HTTPie 

See also

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Powered by Buttondown.