---
title: "Curl"
description: "cURL is a command-line tool for making HTTP requests, transferring data using URLs, and testing APIs. Essential commands for web development and API testing."
author: "Mohammad Abu Mattar"
canonical: https://mkabumattar.com/cheatsheets/curl
---

# Curl

## Getting started with Curl

cURL (client URL) is a command-line tool for transferring data using URLs. It speaks HTTP, HTTPS, FTP, SFTP, and many other protocols, which makes it the usual choice for web development, API testing, and scripting. The common jobs are testing a REST API, downloading a file, and debugging an HTTP connection.

## What makes cURL essential

- Protocols: HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TELNET, LDAP, LDAPS, and more
- Works with any web service or API over those protocols
- Scriptable, which makes requests repeatable in automation
- Verbose output for diagnosing connection and API issues
- No GUI required, and it runs on minimal resources

## Common use cases

```bash
# Test if an API endpoint is responding
curl https://api.github.com/users/github

# Download files
curl -O https://example.com/file.tar.gz

# POST data to a form
curl -X POST -d "username=user&password=pass" https://example.com/login

# Check API response status
curl -I https://api.example.com/health

# Debug API connections
curl -v https://api.example.com/endpoint
```

## Installation quick reference

**macOS**: `brew install curl`
**Ubuntu/Debian**: `sudo apt-get install curl`
**CentOS/RHEL**: `sudo yum install curl`
**Windows**: Download from [curl.se](https://curl.se/download.html) or `choco install curl`

## Essential options reference

| Option                         | Purpose                        |
| ------------------------------ | ------------------------------ |
| `-X GET/POST/PUT/DELETE/PATCH` | Specify HTTP method            |
| `-H "Header: value"`           | Add custom header              |
| `-d 'data'`                    | Send POST data (URL encoded)   |
| `-F "file=@path"`              | Upload file (multipart form)   |
| `-u user:pass`                 | HTTP Basic authentication      |
| `-b "cookie=value"`            | Send cookie                    |
| `-c cookies.txt`               | Save response cookies          |
| `-i`                           | Include response headers       |
| `-I`                           | Headers only (HEAD request)    |
| `-L`                           | Follow redirects               |
| `-v` / `-vv`                   | Verbose output (debug)         |
| `-s`                           | Silent mode (no progress)      |
| `-o file.txt`                  | Save to file                   |
| `-O`                           | Save with original filename    |
| `-w "%{http_code}"`            | Custom output format           |
| `-k` / `--insecure`            | Skip SSL verification (unsafe) |
| `--cacert ca.pem`              | Use custom CA certificate      |
| `--max-time 10`                | Timeout after 10 seconds       |
| `--compressed`                 | Request compression            |

## HTTP methods quick start

```bash
# GET - Retrieve data (default)
curl https://api.example.com/users

# POST - Create resource
curl -X POST -H "Content-Type: application/json" \
  -d '{"name":"John"}' https://api.example.com/users

# PUT - Replace entire resource
curl -X PUT -H "Content-Type: application/json" \
  -d '{"name":"Jane","age":30}' https://api.example.com/users/1

# PATCH - Update specific fields
curl -X PATCH -H "Content-Type: application/json" \
  -d '{"age":31}' https://api.example.com/users/1

# DELETE - Remove resource
curl -X DELETE https://api.example.com/users/1
```

## Workflow examples

### Testing a REST API

```bash
# 1. Test get all users
curl https://api.example.com/users

# 2. Create a new user
curl -X POST -H "Content-Type: application/json" \
  -d '{"name":"Alice","email":"alice@example.com"}' \
  https://api.example.com/users

# 3. Get specific user
curl https://api.example.com/users/1

# 4. Update user
curl -X PATCH -H "Content-Type: application/json" \
  -d '{"email":"alice.new@example.com"}' \
  https://api.example.com/users/1

# 5. Delete user
curl -X DELETE https://api.example.com/users/1
```

### Authenticated API requests

```bash
# Using bearer token
TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/protected

# Using API key
API_KEY="sk_live_abc123xyz789"
curl -H "X-API-Key: $API_KEY" https://api.example.com/data

# Using Basic auth
curl -u username:password https://api.example.com/protected

# OAuth2 flow
TOKEN=$(curl -X POST -d "client_id=xxx&client_secret=yyy&grant_type=client_credentials" https://auth.example.com/token | jq -r '.access_token')
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/data
```

### File handling

```bash
# Download file
curl -o myfile.txt https://example.com/file.txt

# Upload file
curl -F "file=@myfile.txt" https://api.example.com/upload

# Download and extract
curl -s https://example.com/archive.tar.gz | tar xz

# Resume interrupted download
curl -C - -o largefile.iso https://example.com/largefile.iso

# Upload multiple files
curl -F "files=@file1.txt" -F "files=@file2.txt" https://api.example.com/upload-multiple
```

### Debugging and troubleshooting

```bash
# Verbose output (see all request/response details)
curl -v https://api.example.com/data

# More verbose (shows all low-level details)
curl -vv https://api.example.com/data

# Show headers only
curl -i https://api.example.com/data

# Check response time
curl -w "Total time: %{time_total}s\n" https://api.example.com/data

# Save full debug output
curl -v https://api.example.com/data > response.html 2> debug.txt

# Test with custom headers
curl -H "X-Custom: value" -v https://api.example.com/data
```

### Batch processing

```bash
# Check multiple endpoints
for url in api1.com api2.com api3.com; do
  echo -n "$url: "
  curl -s -o /dev/null -w "%{http_code}\n" "https://$url"
done

# Process JSON array response
curl -s https://api.example.com/users | jq -r '.[] | .email' | \
  while read email; do
    echo "Processing: $email"
    curl -X POST -H "Content-Type: application/json" \
      -d "{\"email\":\"$email\",\"status\":\"active\"}" \
      https://api.example.com/notify
  done

# Parallel requests
for i in {1..10}; do
  curl -s "https://api.example.com/data/$i" &
done
wait
echo "All requests completed"
```

## Performance tips

- Use `--compressed` for APIs that support gzip/deflate compression
- Add timeouts with `--max-time` and `--connect-timeout` to prevent hanging
- Batch requests with `&` and `wait` for parallel execution
- Filter responses with `jq` or `grep` to reduce processing time
- Use HEAD requests (`-I`) to check whether a resource exists without downloading it

## Common patterns

**Check if service is up**:

```bash
curl -f -s -o /dev/null -w "%{http_code}\n" https://api.example.com/health
```

**Loop with retry**:

```bash
for attempt in {1..3}; do
  curl -f https://api.example.com/data && break
  [ $attempt -lt 3 ] && sleep 2
done
```

**Extract and transform**:

```bash
curl -s https://api.example.com/users | jq '[.[] | select(.active) | {name, email}]'
```

**Save with authentication**:

```bash
curl -u user:pass -H "Accept: application/json" https://api.example.com/data > data.json
```

## Best practices

1. Use HTTPS for authenticated requests
2. Check exit codes and HTTP status codes
3. Store tokens and credentials in environment variables, not in scripts
4. Check JSON syntax and validate responses before processing
5. Prevent hanging requests with `--max-time` and `--connect-timeout`
6. Redact sensitive data from logs
7. Test API calls against staging before production use
8. Comment curl commands to explain what they do and why

## curl vs other tools

| Tool                | Best For                               |
| ------------------- | -------------------------------------- |
| **curl**            | Command-line, scripts, quick testing   |
| **wget**            | Large file downloads, mirroring sites  |
| **Postman**         | Interactive API testing with GUI       |
| **HTTPie**          | More intuitive syntax, pretty output   |
| **Python requests** | Complex automation and data processing |

## Getting help

- **Manual**: `man curl` or `curl --help`
- **Online**: [curl.se docs](https://curl.se/docs/)
- **Examples**: [Everything curl book](https://ec.haxx.se/everything-curl.pdf)
- **Common Issues**: Check [curl troubleshooting guide](https://curl.se/docs/faq.html)

---

**Pro Tip**: Create a `.curlrc` file in your home directory to store default options and headers for commonly used APIs. This keeps your commands clean and consistent across multiple requests.

## Getting Started

Learn what curl is and make your first HTTP requests

### What is Curl

What curl does and which protocols it handles

**Keywords:** introduction, what, about, curl, http

#### Install curl and check version

```bash
# On macOS with Homebrew
brew install curl

# On Debian/Ubuntu
sudo apt-get install curl

# On CentOS/RHEL
sudo yum install curl

# Check version
curl --version
```

_exec_
```bash
curl --version
```

_output_
```bash
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1
Release-Date: 2023-02-20
```

Installs curl and displays version information showing libcurl version and supported protocols.

- curl is usually pre-installed on most Unix-like systems
- If missing, install via package manager (apt, yum, brew, etc.)
- curl supports HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, TELNET, LDAP, LDAPS, and many more

#### Basic curl command structure

```bash
# Basic syntax
curl [options] [URL]

# Simplest request - GET a URL
curl https://example.com

# Get just the response code
curl -o /dev/null -s -w "%{http_code}" https://example.com

# Show response headers
curl -i https://example.com
```

_exec_
```bash
curl -s -w "%{http_code}\n" https://httpbin.org/get
```

_output_
```bash
200
```

Demonstrates basic curl syntax and how to retrieve HTTP status codes from endpoints.

- By default, curl sends a GET request to the URL
- -s flag enables silent mode (no progress meter)
- -w flag allows custom output formatting with HTTP response codes
- Response is written to stdout unless redirected

#### Quick API testing examples

```bash
# Test if a server is up
curl -I https://api.example.com

# Get response with timing information
curl -w "Response time: %{time_total}s\n" https://example.com

# Test followed by status code only
curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" https://api.example.com
```

_exec_
```bash
curl -I -s https://httpbin.org/get
```

_output_
```bash
HTTP/2 200
Date: Fri, 28 Feb 2025 12:00:00 GMT
Content-Type: application/json
Server: gunicorn/19.9.0
```

Quickly test if APIs are responding without downloading full response body.

- -I flag retrieves headers only (HEAD request)
- Useful for health checks and availability testing
- -w provides detailed timing information

**Best practices:**

- [object Object]
- Use curl for simple requests, scripts, and debugging APIs
- Use verbose mode (-v) when troubleshooting connections

**Common errors:**

- **Could not resolve host**: Check internet connection and hostname spelling
- **Connection timed out**: Verify server is running and firewall allows connections

### HTTP Methods Fundamentals

The HTTP methods and when each one applies

**Keywords:** methods, get, post, put, delete, patch

#### Specify HTTP method with -X flag

```bash
# GET request (default)
curl https://api.example.com/users

# Explicitly specify GET
curl -X GET https://api.example.com/users

# Use different methods
curl -X POST https://api.example.com/users
curl -X PUT https://api.example.com/users/1
curl -X DELETE https://api.example.com/users/1
curl -X PATCH https://api.example.com/users/1
```

_exec_
```bash
curl -X GET https://httpbin.org/get
```

_output_
```bash
{
  "args": {},
  "headers": {
    "Host": "httpbin.org",
    "User-Agent": "curl/7.88.1"
  },
  "origin": "192.0.2.1",
  "url": "https://httpbin.org/get"
}
```

Use the -X flag to explicitly specify the HTTP method for your request.

- GET is the default method (retrieves data)
- POST creates new resources
- PUT replaces entire resource
- DELETE removes resources
- PATCH modifies part of resource

#### Method-specific behavior

```bash
# GET - no body sent
curl https://api.example.com/users?page=1

# POST - requires body
curl -X POST -d "name=John" https://api.example.com/users

# PUT - replace entire resource
curl -X PUT -d '{"name":"Jane","email":"jane@example.com"}' https://api.example.com/users/1

# DELETE - no body, just delete
curl -X DELETE https://api.example.com/users/1
```

_exec_
```bash
curl -X POST -d "key=value" https://httpbin.org/post
```

_output_
```bash
{
  "form": {
    "key": "value"
  },
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  }
}
```

Different HTTP methods handle data and requests differently, with appropriate use cases.

- GET requests are cacheable and idempotent
- POST creates new resources and is not idempotent
- Methods determine server behavior and response

**Best practices:**

- Use correct HTTP method for intended operation
- GET for retrieval, POST for creation, PUT for updates, DELETE for deletion
- Respect API documentation for method requirements

**Common errors:**

- **405 Method Not Allowed**: Check API docs for supported methods on endpoint

### Simple Requests

Make your first requests to APIs and websites

**Keywords:** simple, basic, easy, first, request

#### Get a web page HTML content

```bash
# Fetch HTML from a website
curl https://example.com

# Save HTML to file
curl https://example.com > page.html

# Get with all headers
curl -i https://example.com

# Save to file with proper name
curl -o index.html https://example.com
```

_exec_
```bash
curl -s https://httpbin.org/html | head -20
```

_output_
```bash
<!DOCTYPE html>
<html>
  <head>
  </head>
  <body>
      <h1>Herman Melville - Moby-Dick</h1>
```

Fetch web content using curl and save or display the response.

- Without flags, response is printed to stdout
- Use > to redirect to file or -o to save
- -i includes HTTP headers in response

#### Query parameters and URLs

```bash
# Simple URL with query parameters
curl "https://api.example.com/search?q=curl&page=1"

# Handle special characters in parameters
curl "https://api.example.com/search?q=hello%20world"

# URL encoding alternative
curl --data-urlencode "q=hello world" https://api.example.com/search

# Test with httpbin
curl "https://httpbin.org/get?name=John&age=30"
```

_exec_
```bash
curl -s "https://httpbin.org/get?name=Alice&age=25" | grep -A5 args
```

_output_
```bash
"args": {
  "age": "25",
  "name": "Alice"
}
```

Pass query parameters to APIs using URL query strings.

- Use quotes around URLs with special characters
- Query parameters start with ?
- Multiple parameters separated by &
- URL encode special characters

#### Follow redirects

```bash
# Follow HTTP redirects
curl -L https://example.com

# Follow max 5 redirects
curl -L --max-redirs 5 https://example.com

# Show redirect chain
curl -L -v https://example.com 2>&1 | grep ">"
```

_exec_
```bash
curl -L -s -w "Final URL: %{url_effective}\n" https://httpbin.org/redirect/2
```

_output_
```bash
Final URL: https://httpbin.org/relative-redirect/1
```

Use -L flag to automatically follow HTTP redirects (301, 302, etc.).

- Default curl doesn't follow redirects
- -L (--location) enables automatic redirect following
- --max-redirs limits maximum redirects to prevent loops

**Best practices:**

- Always use quotes around URLs with special characters
- Use -L when testing public APIs to handle redirects
- Save responses to files for analysis

**Common errors:**

- **Empty response or just headers**: Check if page redirects (-L flag helps)

## Request Methods

Detailed usage of different HTTP request methods

### GET Requests

Retrieve data from servers with GET requests

**Keywords:** get, retrieve, fetch, query

#### Basic GET requests with parameters

```bash
# Get with parameters
curl https://api.example.com/users?limit=10&offset=0

# Get specific resource
curl https://api.example.com/users/123

# Get with multiple parameters
curl "https://api.example.com/posts?author=john&status=published&sort=date"

# Use -G to send POST data as GET
curl -G https://api.example.com/search -d "q=api" -d "limit=10"
```

_exec_
```bash
curl -s "https://httpbin.org/get?id=1&name=test" | python3 -m json.tool
```

_output_
```bash
{
  "args": {
    "id": "1",
    "name": "test"
  },
  "url": "https://httpbin.org/get?id=1&name=test"
}
```

GET requests retrieve data with parameters appended to the URL.

- GET parameters are visible in URL
- Use for non-sensitive data
- Parameters passed via query string

#### Conditional GET requests

```bash
# Get only if modified since date (caching)
curl -H "If-Modified-Since: Mon, 12 Dec 2022 13:00:00 GMT" https://api.example.com/data

# Check if resource exists (HEAD request for headers only)
curl -I https://api.example.com/users/123

# Get with timeout
curl --max-time 5 https://api.example.com/data
```

_exec_
```bash
curl -I -s "https://httpbin.org/get" | head -5
```

_output_
```bash
HTTP/1.1 200 OK
Date: Fri, 28 Feb 2025 12:00:00 GMT
Content-Type: application/json
Content-Length: 347
```

Optimize GET requests with conditional headers and timeouts.

- -I gets headers only, skipping the body
- --max-time prevents hanging on slow servers
- If-Modified-Since improves caching

#### Handle large responses

```bash
# Get large file and show progress
curl -# https://api.example.com/large-dataset.json -o data.json

# Resume interrupted download
curl -C - -o data.json https://api.example.com/large-file.json

# Stream response (don't wait for completion)
curl -N https://api.example.com/stream

# Get with progress bar
curl -# -O https://example.com/file.zip
```

_exec_
```bash
curl -s "https://httpbin.org/get" | wc -c
```

_output_
```bash
389
```

Download large files with progress tracking and resume support.

- -# shows simple progress bar
- -C - resumes partial downloads
- -N enables streaming responses
- -O saves with original filename

**Best practices:**

- Use query parameters for filtering and pagination
- Always quote URLs with parameters
- Use -I flag for HEAD requests when only checking headers

**Common errors:**

- **400 Bad Request**: Check parameter syntax and URL encoding

### POST Requests

Send data to servers with POST requests

**Keywords:** post, create, submit, data

#### POST with form data

```bash
# POST with form data (URL encoded)
curl -X POST -d "name=John&email=john@example.com" https://api.example.com/users

# POST with multiple form fields
curl -d "user=john" -d "password=secret" -d "login=true" https://api.example.com/login

# Post data from file
curl -d @data.txt https://api.example.com/submit

# Alternative form data syntax
curl -X POST --data "key1=value1&key2=value2" https://api.example.com/create
```

_exec_
```bash
curl -s -X POST -d "user=john&age=30" https://httpbin.org/post | python3 -m json.tool
```

_output_
```bash
{
  "form": {
    "age": "30",
    "user": "john"
  }
}
```

Send form data using POST requests with URL-encoded parameters.

- -d automatically sets Content-Type to application/x-www-form-urlencoded
- Multiple -d flags combine parameters
- @filename sends file contents as data

#### POST with JSON data

```bash
# POST JSON data
curl -X POST -H "Content-Type: application/json" \
  -d '{"name":"John","email":"john@example.com"}' \
  https://api.example.com/users

# POST JSON from file
curl -X POST -H "Content-Type: application/json" \
  -d @user.json https://api.example.com/users

# Using jq to format JSON
curl -X POST -H "Content-Type: application/json" \
  -d "$(echo '{"name":"John"}' | jq .)" \
  https://api.example.com/users
```

_exec_
```bash
curl -s -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://httpbin.org/post | python3 -m json.tool
```

_output_
```bash
{
  "json": {
    "key": "value"
  }
}
```

Send JSON data using proper Content-Type header to APIs expecting JSON.

- Must set Content-Type: application/json header
- Payload sent as JSON string
- Use single quotes to preserve JSON structure

#### POST with file upload

```bash
# Upload file with multipart form
curl -F "file=@image.png" https://api.example.com/upload

# Upload with additional fields
curl -F "file=@photo.jpg" -F "title=My Photo" -F "description=A beautiful photo" https://api.example.com/upload

# Upload multiple files
curl -F "files=@file1.txt" -F "files=@file2.txt" https://api.example.com/upload-multiple

# Upload with HTTP Basic auth
curl -F "file=@document.pdf" -u user:pass https://api.example.com/upload
```

_exec_
```bash
curl -s -F "file=@/etc/hosts" -F "title=test" https://httpbin.org/post
```

_output_
```bash
{
  "files": {
    "file": "[binary content]"
  },
  "form": {
    "title": "test"
  }
}
```

Upload files using multipart/form-data encoding with optional additional fields.

- -F flag sets Content-Type to multipart/form-data automatically
- @ symbol indicates file path
- Can combine multiple files and form fields

**Best practices:**

- Use JSON for APIs, form data for HTML forms
- Always set proper Content-Type header
- Send sensitive data via POST, not GET

**Common errors:**

- **415 Unsupported Media Type**: Check Content-Type header matches API requirements

### PUT, PATCH, and DELETE Requests

Update and delete resources with different methods

**Keywords:** put, patch, delete, update, remove

#### PUT and PATCH requests

```bash
# PUT - Replace entire resource
curl -X PUT -H "Content-Type: application/json" \
  -d '{"name":"Jane","email":"jane@example.com","age":28}' \
  https://api.example.com/users/1

# PATCH - Update specific fields only
curl -X PATCH -H "Content-Type: application/json" \
  -d '{"email":"newemail@example.com"}' \
  https://api.example.com/users/1

# PUT with form data
curl -X PUT -d "name=John&status=active" https://api.example.com/users/1

# PATCH with file
curl -X PATCH -H "Content-Type: application/json" \
  -d @user-update.json https://api.example.com/users/1
```

_exec_
```bash
curl -s -X PATCH -H "Content-Type: application/json" -d '{"status":"updated"}' https://httpbin.org/patch | python3 -m json.tool
```

_output_
```bash
{
  "json": {
    "status": "updated"
  }
}
```

Use PUT to replace entire resources and PATCH to partially update them.

- PUT replaces entire resource (must provide all fields)
- PATCH updates only specified fields
- Both typically require authentication

#### DELETE requests

```bash
# Simple DELETE
curl -X DELETE https://api.example.com/users/1

# DELETE with response verification
curl -X DELETE -w "Status: %{http_code}\n" https://api.example.com/users/1

# DELETE with authorization
curl -X DELETE -H "Authorization: Bearer token123" https://api.example.com/users/1

# DELETE with query parameters
curl -X DELETE "https://api.example.com/posts?id=123&confirm=true"
```

_exec_
```bash
curl -X DELETE -w "\nHTTP Status: %{http_code}\n" https://httpbin.org/delete
```

_output_
```bash
HTTP Status: 200
```

Remove resources from the server using DELETE requests.

- DELETE is idempotent (multiple calls have same effect)
- Typically returns 204 (No Content) or 200 (OK)
- May require authentication

#### Request with body and status verification

```bash
# Update and verify response
curl -X PUT \
  -H "Content-Type: application/json" \
  -d '{"status":"active"}' \
  -w "\nStatus: %{http_code}\nTime: %{time_total}s\n" \
  https://api.example.com/users/1

# Delete and check status
curl -i -X DELETE https://api.example.com/users/1

# Save response to file on successful update
curl -X PATCH -d "status=completed" \
  -o response.json \
  -w "%{http_code}" \
  https://api.example.com/tasks/1
```

_exec_
```bash
curl -i -X DELETE https://httpbin.org/delete | head -10
```

_output_
```bash
HTTP/2 200
Date: Fri, 28 Feb 2025 12:00:00 GMT
Content-Type: application/json
```

Combine requests with response verification and output handling.

- -i includes headers and response
- -w allows custom output formatting
- Combine with -o to save to file

**Best practices:**

- Use PUT for complete replacement, PATCH for partial updates
- Always verify status codes for update operations
- Handle 409 (Conflict) and 404 (Not Found) errors

**Common errors:**

- **404 Not Found**: Verify resource ID is correct
- **409 Conflict**: Resource may have been modified, try refreshing

## Headers & Authentication

Work with HTTP headers and authentication mechanisms

### Custom Headers

Add custom headers to requests

**Keywords:** headers, custom, user-agent, referer

#### Set custom headers with -H flag

```bash
# Single header
curl -H "X-API-Key: abc123" https://api.example.com/data

# Multiple headers
curl -H "X-API-Key: abc123" -H "X-Request-ID: req-123" https://api.example.com/data

# Content-Type for JSON
curl -H "Content-Type: application/json" \
  -d '{"key":"value"}' https://api.example.com/submit

# Accept header for response format
curl -H "Accept: application/json" https://api.example.com/data
```

_exec_
```bash
curl -s -H "X-Custom: hello" https://httpbin.org/headers | python3 -m json.tool
```

_output_
```bash
{
  "headers": {
    "X-Custom": "hello",
    "User-Agent": "curl/7.88.1"
  }
}
```

Add custom headers to requests using the -H flag (multiple times if needed).

- Format: -H "Header-Name: Header-Value"
- Multiple headers require multiple -H flags
- Headers are case-insensitive

#### User-Agent and Referer headers

```bash
# Set user agent (identify as browser)
curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" https://example.com

# Alternative syntax for user agent
curl -H "User-Agent: MyApp/1.0" https://api.example.com/data

# Set referer (HTTP_REFERER)
curl -e "https://google.com" https://example.com

# Combined user agent and referer
curl -A "Mozilla/5.0" -e "https://google.com" https://example.com
```

_exec_
```bash
curl -s -A "CustomBot/1.0" https://httpbin.org/user-agent
```

_output_
```bash
{
  "user-agent": "CustomBot/1.0"
}
```

Set User-Agent and Referer headers to identify your request origin and client.

- -A is shortcut for setting User-Agent
- -e sets Referer header (note spelling)
- Some sites may block requests without proper User-Agent

#### Request header inspection

```bash
# View all request headers sent
curl -v https://api.example.com/data 2>&1 | grep ">"

# Show comprehensive header information
curl -v -H "Authorization: Bearer token" https://api.example.com/data 2>&1 | head -20

# Test with custom headers and see what was sent
curl -H "X-Test: value1" -H "X-Test2: value2" -v https://httpbin.org/headers 2>&1 | grep ">"
```

_exec_
```bash
curl -v https://httpbin.org/headers 2>&1 | grep ">"
```

_output_
```bash
> GET /headers HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.88.1
> Accept: */*
```

View outgoing request headers using -v (verbose) flag.

- Lines starting with > show request headers
- Lines starting with < show response headers
- 2>&1 redirects stderr to stdout

**Best practices:**

- Always set Content-Type for requests with body
- Use Accept header to specify desired response format
- Set User-Agent when scraping or testing web services

**Common errors:**

- **415 Unsupported Media Type**: Match the Content-Type header to the data format

### Authentication Methods

Authenticate requests with various methods

**Keywords:** auth, authentication, basic, bearer, token

#### HTTP Basic Authentication

```bash
# Basic auth with -u flag
curl -u username:password https://api.example.com/protected

# Basic auth alternative with header
curl -H "Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=" https://api.example.com/protected

# Base64 encode credentials
echo -n "username:password" | base64

# Prompt for password (don't show in history)
curl -u username https://api.example.com/protected
```

_exec_
```bash
curl -s -u "user:pass" https://httpbin.org/basic-auth/user/pass
```

_output_
```bash
{
  "authenticated": true,
  "user": "user"
}
```

Use HTTP Basic Authentication with -u flag (username:password).

- Credentials are Base64 encoded, not encrypted
- Always use HTTPS with Basic Auth (not HTTP)
- -u without password prompts for it securely

#### Bearer token and API key authentication

```bash
# Bearer token (OAuth2, JWT, etc)
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIs..." https://api.example.com/data

# API key in header
curl -H "X-API-Key: sk_live_abc123xyz789" https://api.example.com/data

# API key as query parameter
curl "https://api.example.com/data?api_key=sk_live_abc123"

# Store token in variable and use it
TOKEN="eyJhbGciOiJIUzI1NiIs..."
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/data
```

_exec_
```bash
curl -s -H "Authorization: Bearer test-token" https://httpbin.org/bearer | python3 -m json.tool
```

_output_
```bash
{
  "authenticated": true,
  "token": "test-token"
}
```

Authenticate with bearer tokens (JWT, OAuth2) and API keys.

- Bearer tokens include "Bearer " prefix
- API keys are commonly passed in headers or query params
- Tokens are sensitive, avoid including in logs

#### Complex authentication scenarios

```bash
# Digest authentication (more secure than basic)
curl --digest -u username:password https://api.example.com/protected

# Get JWT token then use it
TOKEN=$(curl -s -X POST -d "user=john&pass=secret" https://api.example.com/auth | jq -r '.token')
curl -H "Authorization: Bearer $TOKEN" https://api.example.com/data

# Use netrc file for credentials (~/.netrc)
curl --netrc https://api.example.com/protected

# Pass credentials in URL (deprecated, use -u instead)
curl "https://user:pass@api.example.com/protected"
```

_exec_
```bash
curl -s --digest -u "user:passwd" https://httpbin.org/digest-auth/auth/user/passwd
```

_output_
```bash
{
  "authenticated": true,
  "user": "user"
}
```

Implement more advanced authentication flows and secure credential handling.

- Digest auth is more secure than Basic auth
- Never include passwords in URLs
- Use environment variables or .netrc for sensitive credentials

**Best practices:**

- Always use HTTPS for authenticated requests
- Store tokens in environment variables or secure vaults
- Use -u without password to avoid shell history

**Common errors:**

- **401 Unauthorized**: Check credentials and token expiration

### Cookies & Sessions

Handle cookies and maintain sessions

**Keywords:** cookies, session, cache, maintain

#### Set and send cookies

```bash
# Send cookie with request
curl -b "session=abc123" https://api.example.com/data

# Multiple cookies
curl -b "session=abc123" -b "user_id=456" https://api.example.com/data

# Cookie from file
curl -b cookies.txt https://api.example.com/data

# Send raw cookie string
curl -b "name1=value1; name2=value2" https://example.com
```

_exec_
```bash
curl -s -b "test=value123" https://httpbin.org/cookies | python3 -m json.tool
```

_output_
```bash
{
  "cookies": {
    "test": "value123"
  }
}
```

Send cookies with requests using -b flag.

- -b sends cookies, -B sends from file, -c saves to file
- Multiple -b flags combine cookies
- Cookie format: name=value

#### Save and load cookies

```bash
# Save response cookies to file
curl -c cookies.txt https://api.example.com/login

# Load cookies from file for subsequent requests
curl -b cookies.txt https://api.example.com/protected

# Save and load in single command
curl -c cookies.txt -b cookies.txt https://api.example.com/data

# Enable cookie jar for session persistence
curl -c ~/.curl_cookies -b ~/.curl_cookies https://api.example.com/login
```

_exec_
```bash
curl -s https://httpbin.org/cookies/set?test=value123 -c /tmp/cookies.txt && cat /tmp/cookies.txt
```

_output_
```bash
# Netscape HTTP Cookie File
.httpbin.org	TRUE	/	FALSE	0	test	value123
```

Save cookies from responses and reuse them in subsequent requests for session management.

- -c saves response cookies to file
- Netscape HTTP Cookie File format is standard
- Useful for maintaining sessions across multiple requests

#### Session workflow example

```bash
# Typical workflow: login, then use session

# 1. Login and save session cookie
curl -c cookies.txt -X POST \
  -d "username=john&password=secret" \
  https://api.example.com/login

# 2. Use session for authenticated requests
curl -b cookies.txt https://api.example.com/profile

# 3. Make multiple requests with same session
curl -b cookies.txt https://api.example.com/posts
curl -b cookies.txt -X POST -d "content=hello" https://api.example.com/posts

# 4. Logout and clear cookies
curl -b cookies.txt -X POST https://api.example.com/logout
rm cookies.txt
```

_exec_
```bash
curl -s -c - https://httpbin.org/cookies/set?session=xyz123 | grep -i httpbin
```

_output_
```bash
.httpbin.org	TRUE	/	FALSE	0	session	xyz123
```

Manage complete session workflow from login to authenticated requests to logout.

- Always save and load cookies for stateful APIs
- Use same cookie jar across related requests
- Clear cookies after logout for security

**Best practices:**

- Use cookie files for session persistence in scripts
- Keep cookies file secure if contains sensitive data
- Clear cookies when session ends

**Common errors:**

- **403 Forbidden after login**: Verify cookies were saved and loaded properly

## Data Transfer

Handle different data formats and file operations

### JSON Data

Work with JSON data in requests and responses

**Keywords:** json, data, api, format

#### Send JSON requests

```bash
# Basic JSON POST
curl -X POST -H "Content-Type: application/json" \
  -d '{"name":"John","email":"john@example.com"}' \
  https://api.example.com/users

# Pretty JSON with jq
curl -X POST -H "Content-Type: application/json" \
  -d "$(jq -n --arg name "John" --arg email "john@example.com" '{name: $name, email: $email}')" \
  https://api.example.com/users

# Nested JSON
curl -X POST -H "Content-Type: application/json" \
  -d '{"user":{"name":"John","address":{"city":"NYC"}},"active":true}' \
  https://api.example.com/users
```

_exec_
```bash
curl -s -X POST -H "Content-Type: application/json" -d '{"test":"data"}' https://httpbin.org/post | python3 -m json.tool
```

_output_
```bash
{
  "json": {
    "test": "data"
  }
}
```

Send properly formatted JSON data to APIs.

- Always set Content-Type: application/json
- Single quotes preserve special characters in JSON
- Escape quotes in JSON: \" for double quotes

#### Parse and modify JSON responses

```bash
# Get JSON and parse with jq
curl -s https://api.example.com/users | jq '.'

# Extract specific field
curl -s https://api.example.com/users | jq '.users[0].name'

# Filter and map responses
curl -s https://api.example.com/users | jq '[.users[] | select(.active) | {name, email}]'

# Modify response and send as next request
curl -s https://api.example.com/users/1 | jq '.status = "updated"' | \
  curl -X PATCH -H "Content-Type: application/json" -d @- https://api.example.com/users/1
```

_exec_
```bash
curl -s https://httpbin.org/json | python3 -m json.tool | head -10
```

_output_
```bash
{
  "slideshow": {
    "author": "Yours Truly",
    "date": "2003-05-20",
    "slides": [
      {
        "title": "Wake up to WonderWidgets!"
      }
    ]
  }
}
```

Parse, filter, and manipulate JSON responses using jq for complex data operations.

- Install jq: apt-get install jq (or brew install jq on macOS)
- jq handles JSON transformation
- (hyphen) in curl -d @- reads from stdin

#### JSON to form data conversion

```bash
# Convert JSON to form data
curl -X POST -H "Content-Type: application/json" \
  -d '{"username":"john","password":"secret"}' \
  https://api.example.com/api-submit

# Compare with form data version
curl -X POST \
  -d "username=john&password=secret" \
  https://api.example.com/form-submit

# Use correct Content-Type based on API requirements
# JSON APIs need: application/json
# Form APIs need: application/x-www-form-urlencoded
# Multipart APIs need: multipart/form-data (auto with -F)
```

_exec_
```bash
curl -s -X POST -d "key=value&test=data" https://httpbin.org/post | python3 -m json.tool | grep -A5 form
```

_output_
```bash
"form": {
  "key": "value",
  "test": "data"
}
```

Choose correct Content-Type based on API requirements.

- JSON uses application/json
- Form data uses application/x-www-form-urlencoded (default -d)
- Multipart forms use multipart/form-data (-F)

**Best practices:**

- Always validate JSON syntax before sending
- Use jq for complex JSON transformations
- Set Content-Type explicitly for clarity

**Common errors:**

- **Invalid JSON**: Use jq to validate syntax before sending

### File Uploads & Downloads

Upload and download files with curl

**Keywords:** upload, download, file, transfer

#### Upload files with multipart form

```bash
# Upload single file
curl -F "file=@/path/to/file.txt" https://api.example.com/upload

# Upload with additional metadata
curl -F "file=@/path/to/image.png" \
  -F "title=My Image" \
  -F "description=Beautiful image" \
  https://api.example.com/upload

# Upload multiple files
curl -F "files=@file1.txt" -F "files=@file2.txt" -F "files=@file3.txt" \
  https://api.example.com/upload-batch

# Upload with authentication
curl -F "file=@document.pdf" -u user:pass https://api.example.com/upload
```

_exec_
```bash
curl -s -F "file=@/etc/hostname" https://httpbin.org/post | python3 -m json.tool | head -15
```

_output_
```bash
{
  "files": {
    "file": "[binary content]"
  }
}
```

Upload files with multipart form encoding for APIs that accept file uploads.

- -F flag automatically sets multipart/form-data
- @ prefix indicates file path
- Repeat -F for multiple files with same field name

#### Download and resume downloads

```bash
# Download file
curl -o filename.zip https://example.com/large-file.zip

# Download with original filename
curl -O https://example.com/file.pdf

# Resume interrupted download
curl -C - -o file.zip https://example.com/file.zip

# Show download progress
curl -# -o file.zip https://example.com/file.zip

# Download multiple files
curl -o file1.txt https://example.com/file1.txt -o file2.txt https://example.com/file2.txt
```

_exec_
```bash
curl -s -I https://httpbin.org/image/jpeg | grep Content-Length
```

_output_
```bash
Content-Length: 8633
```

Download files with progress tracking and resume support.

- -o specifies output filename
- -O uses original filename from URL
- -C - resumes partial downloads
- # shows simple progress bar

#### Advanced file operations

```bash
# Download and pipe to another command
curl -s https://example.com/archive.tar.gz | tar xz

# Download multiple files from listing
curl -s https://example.com/files.json | jq -r '.[] | .url' | \
  xargs -I {} curl -O {}

# Upload file with progress and error handling
if curl -F "file=@largefile.bin" \
  --progress-bar \
  https://api.example.com/upload; then
  echo "Upload successful"
else
  echo "Upload failed"
fi

# Download with custom headers and authentication
curl -H "Authorization: Bearer token" \
  -H "Accept: application/octet-stream" \
  -o downloaded-file \
  https://api.example.com/download
```

_exec_
```bash
curl -s https://httpbin.org/image/png | file -
```

_output_
```bash
standard input: PNG image data, 200 x 200...
```

Combine curl with other tools for complex file operations and error handling.

- Pipe output to tar, unzip, etc. for automated extraction
- Use xargs to download multiple files in parallel
- Check status codes ($?) for error handling

**Best practices:**

- Use -C - to resume interrupted downloads
- Show progress with
- Always verify file integrity after download if critical

**Common errors:**

- **Permission denied when uploading**: Check file permissions and API authorization

### Streaming & Chunked Data

Handle streaming and chunked data transfer

**Keywords:** streaming, chunks, large, progressive

#### Stream responses

```bash
# Stream without buffering (don't wait for complete response)
curl -N https://api.example.com/stream

# Process streaming JSON lines
curl -N https://example.com/events | while read line; do
  echo "$line" | jq '.event'
done

# Stream with timeout
curl -N --max-time 30 https://api.example.com/stream

# Save stream content
curl -N https://example.com/stream -o stream.log
```

_exec_
```bash
timeout 2 curl -s -N https://httpbin.org/stream-bytes/100 | xxd | head -5
```

_output_
```bash
00000000: 8d5e 16cf 9cac 098e e3dc 1fa3 5f7e 9bb7  .^.........._~..
00000010: 89e9 7c5a ed66 7fa5 e901 40b5 0b46 6903  ..|Z.f....@..Fi.
```

Receive streaming responses and process them line-by-line without waiting.

- -N enables streaming (no buffering)
- Useful for long-lived streams and event feeds
- Can save to file or pipe to other commands

#### Chunked uploads

```bash
# Upload in chunks (for large files)
# Using HTTP/1.1 Transfer-Encoding: chunked
curl -H "Transfer-Encoding: chunked" \
  -d @largefile.bin \
  https://api.example.com/upload

# Stream file content
curl -X PUT -H "Content-Type: application/octet-stream" \
  --data-binary @file.bin \
  https://api.example.com/objects/file.bin

# Split large file and upload parts
split -b 1M largefile.bin part_
for part in part_*; do
  curl -X POST -F "file=@$part" \
    https://api.example.com/upload-part
done
```

_exec_
```bash
echo "test data" | curl -s -X POST -d @- https://httpbin.org/post
```

_output_
```bash
{
  "data": "test data"
}
```

Upload large files in chunks or streams instead of buffering them whole.

- Transfer-Encoding: chunked for chunked uploads
- --data-binary preserves binary file content
- Split and reassemble for multi-part uploads

**Best practices:**

- Use streaming for real-time data and large files
- Add timeout to prevent hanging streams
- Process chunked data immediately to save memory

**Common errors:**

- **premature end of stream**: Check connection stability, add timeout

## Output & Debugging

Inspect responses and debug requests

### Verbose Mode & Inspection

Debug requests and responses with verbose output

**Keywords:** verbose, debug, headers, inspect

#### Enable verbose mode for troubleshooting

```bash
# Basic verbose mode (shows headers)
curl -v https://api.example.com/data

# Extra verbose (shows all details)
curl -vv https://api.example.com/data

# Verbose to file
curl -v https://api.example.com/data > response.html 2> debug.txt

# Show request headers only
curl -v https://api.example.com/data 2>&1 | grep ">"

# Show response headers only
curl -v https://api.example.com/data 2>&1 | grep "<"
```

_exec_
```bash
curl -v https://httpbin.org/get 2>&1 | head -15
```

_output_
```bash
*   Trying 34.201.40.105:443...
* Connected to httpbin.org (34.201.40.105) port 443 (#0)
* ALPN, offering h2
> GET /get HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.88.1
```

Use verbose mode to see all request and response details for debugging.

- Lines starting with > are request headers
- Lines starting with < are response headers
- Lines starting with * are curl debug info

#### Show headers without body

```bash
# Headers only (no body)
curl -i https://api.example.com/data

# Headers from HEAD request only
curl -I https://api.example.com/data

# Just response headers with body as well
curl -D - https://api.example.com/data

# Dump headers to file
curl -D headers.txt https://api.example.com/data
```

_exec_
```bash
curl -I -s https://httpbin.org/json 2>&1
```

_output_
```bash
HTTP/2 200
date: Fri, 28 Feb 2025 12:00:00 GMT
content-type: application/json
content-length: 429
```

Inspect response headers without downloading body content.

- -i includes headers and body
- -I gets headers only (HEAD request)
- -D saves headers to file

#### Detailed timing and trace information

```bash
# Show timing breakdown
curl -w "@curl-format.txt" -o /dev/null https://api.example.com/data

# Simple timing format
curl -w "\nTime: %{time_total}s\nSpeed: %{speed_download} bytes/s\n" -o /dev/null https://example.com

# Comprehensive timing
curl -w "\n\nDNS Lookup: %{time_namelookup}s\nTCP Connect: %{time_connect}s\nFirst Byte: %{time_starttransfer}s\nTotal Time: %{time_total}s\n" https://api.example.com/data

# Trace connection information
curl --trace-ascii debug.txt https://api.example.com/data
```

_exec_
```bash
curl -w "\nStatus: %{http_code}\nTime: %{time_total}s\n" -o /dev/null -s https://httpbin.org/get
```

_output_
```bash
Status: 200
Time: 0.452341s
```

Measure performance metrics and timing information for API requests.

- %{time_total} total request time
- %{time_namelookup} DNS resolution time
- %{http_code} HTTP response status code
- --trace-ascii for detailed debugging

**Best practices:**

- Use -v to understand request/response flow
- Check timing with -w for performance issues
- Redirect stderr (2>&1) to capture verbose output

**Common errors:**

- **SSL certificate problem**: Try -k for insecure or --cacert for custom certificate

### Filter & Customize Output

Filter responses and customize output format

**Keywords:** output, filter, grep, sed

#### Extract specific data from responses

```bash
# Extract JSON field with jq
curl -s https://api.example.com/users | jq '.users[0].email'

# Extract HTML content with grep
curl -s https://example.com | grep -oP '(?<=<title>)[^<]*'

# Extract HTTP header value
curl -s -I https://api.example.com/data | grep -i "content-type"

# Count items in response
curl -s https://api.example.com/items | jq '.items | length'
```

_exec_
```bash
curl -s https://httpbin.org/get | python3 -c "import json, sys; data=json.load(sys.stdin); print(data['url'])"
```

_output_
```bash
https://httpbin.org/get
```

Extract specific fields from API responses using jq, grep, and other tools.

- jq best for JSON
- grep with -oP for regex extraction
- pipe | allows chaining commands

#### Format output with -w flag

```bash
# Status code only
curl -w "%{http_code}\n" -o /dev/null https://api.example.com/data

# URL and status
curl -w "%{url_effective}\t%{http_code}\n" -o /dev/null https://api.example.com

# Multiple values separated by tabs
curl -w "%{http_code}\t%{time_total}\t%{size_download}\n" -o /dev/null https://api.example.com/data

# Custom format for batch processing
curl -w "%{filename_effective},%{http_code},%{time_total}\n" -o data.json https://api.example.com/data
```

_exec_
```bash
curl -s -w "\nURL: %{url_effective}\nStatus: %{http_code}\nSize: %{size_download} bytes\n" https://httpbin.org/get -o /dev/null
```

_output_
```bash
URL: https://httpbin.org/get
Status: 200
Size: 389 bytes
```

Customize output format using -w flag for scripting and batch operations.

- %{http_code} HTTP status code
- %{time_total} total time
- %{size_download} downloaded size

#### Batch processing and response validation

```bash
# Check multiple endpoints
for url in https://api.example.com/users https://api.example.com/posts https://api.example.com/comments; do
  status=$(curl -s -o /dev/null -w "%{http_code}" "$url")
  echo "$url: $status"
done

# Validate JSON response
response=$(curl -s https://api.example.com/data)
if echo "$response" | jq empty 2>/dev/null; then
  echo "Valid JSON"
else
  echo "Invalid JSON: $response"
fi

# Extract and process multiple values
curl -s https://api.example.com/users | jq -r '.[] | "ID: \(.id), Name: \(.name), Email: \(.email)"'
```

_exec_
```bash
curl -s https://httpbin.org/get | python3 -c "import json, sys; json.load(sys.stdin); print('Valid JSON')"
```

_output_
```bash
Valid JSON
```

Validate responses and process multiple items from API responses.

- Use jq empty to validate JSON without output
- Process arrays with jq .[]
- Combine with bash loops for batch operations

**Best practices:**

- Use jq for JSON filtering and transformation
- Validate responses before processing
- Use -w for custom output in scripts

**Common errors:**

- **jq parse error**: Check that the response is valid JSON

### Error Handling & Status

Handle errors and check response status

**Keywords:** errors, status, codes, handling

#### Check HTTP status codes

```bash
# Get HTTP status code
status=$(curl -s -o /dev/null -w "%{http_code}" https://api.example.com/data)

# Check if successful (2xx)
if [ $status -ge 200 ] && [ $status -lt 300 ]; then
  echo "Success: $status"
elif [ $status -ge 400 ] && [ $status -lt 500 ]; then
  echo "Client error: $status"
else
  echo "Server error: $status"
fi

# Fail on non-2xx status
curl -f https://api.example.com/data || echo "Request failed"
```

_exec_
```bash
curl -f -s https://httpbin.org/status/404 2>&1 || echo "Exit code: $?"
```

_output_
```bash
Exit code: 22
```

Extract and evaluate HTTP status codes for error handling.

- -f flag returns non-zero exit on HTTP errors
- 2xx = success, 4xx = client error, 5xx = server error
- Check $? for curl exit code

#### Retry logic for failed requests

```bash
# Simple retry with exponential backoff
retry_count=0
max_retries=3
while [ $retry_count -lt $max_retries ]; do
  if curl -f https://api.example.com/data; then
    echo "Success"
    break
  fi
  retry_count=$((retry_count + 1))
  sleep $((2 ** retry_count))
done

# Retry on timeout
curl --connect-timeout 5 --max-time 10 https://api.example.com/data || \
  (sleep 2 && curl --connect-timeout 5 --max-time 10 https://api.example.com/data)

# Retry with custom exit code check
for attempt in {1..3}; do
  curl -f https://api.example.com/data && break
  [ $attempt -lt 3 ] && sleep $((attempt * 2))
done
```

_exec_
```bash
curl -s --connect-timeout 2 https://httpbin.org/delay/1 | head -20
```

_output_
```bash
{
  "args": {},
  "delay": 1,
  "headers": {
```

Implement retry logic for handling transient failures.

- --connect-timeout limits connection time
- --max-time limits total request time
- Exponential backoff prevents overwhelming server

#### Capture errors and responses

```bash
# Capture both success and error responses
response=$(curl -s -w "\nSTATUS:%{http_code}" https://api.example.com/data)
body=$(echo "$response" | sed '$d')
status=$(echo "$response" | tail -1 | cut -d: -f2)

if [ "$status" -eq 200 ]; then
  echo "Success response: $body"
else
  echo "Error ($status): $body"
fi

# Save error responses
curl -s -w "%{http_code}" https://api.example.com/data -o response.json
if [ $? -ne 0 ]; then
  echo "Curl failed" >&2
  cat response.json >&2
  exit 1
fi
```

_exec_
```bash
curl -s -w "\n%{http_code}" https://httpbin.org/get | tail -1
```

_output_
```bash
200
```

Capture status codes and error messages for proper error handling.

- Use -w to append status code to output
- sed '$d' removes last line
- tail -1 gets last line
- check $? for curl exit code

**Best practices:**

- Always check status codes in scripts
- Implement retry logic for transient failures
- Log errors with context for debugging

**Common errors:**

- **curl fails silently**: Use -f flag to exit on HTTP errors

## SSL/TLS & Security

Configure SSL/TLS and handle secure connections

### SSL Certificates

Work with SSL/TLS certificates and HTTPS

**Keywords:** ssl, tls, https, certificate

#### Handle SSL certificate issues

```bash
# Ignore SSL certificate validation (not recommended)
curl -k https://self-signed-cert.example.com

# Use custom CA certificate
curl --cacert /path/to/ca-cert.pem https://api.example.com/data

# Specify client certificate
curl --cert client-cert.pem --key client-key.pem https://api.example.com/data

# Use PKCS12 certificate file
curl --cert certificate.p12:password https://api.example.com/data
```

_exec_
```bash
curl -I https://httpbin.org 2>&1 | head -5
```

_output_
```bash
HTTP/2 200
date: Fri, 28 Feb 2025 12:00:00 GMT
content-type: application/json
```

Configure certificate handling for various HTTPS scenarios.

- -k ignores certificate verification (unsafe)
- --cacert uses custom CA certificate
- --cert specifies client certificate

#### View certificate information

```bash
# Show certificate details
echo | openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -text

# Extract certificate from curl verbose
curl -v https://api.example.com/data 2>&1 | grep "subject:"

# Save certificate to file
echo | openssl s_client -connect api.example.com:443 -showcerts | \
  openssl x509 -out server-cert.pem

# Check certificate expiration
curl --insecure -I https://api.example.com 2>&1 | head -5 && \
  echo | openssl s_client -connect api.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates
```

_exec_
```bash
echo | openssl s_client -connect httpbin.org:443 -showcerts 2>/dev/null | grep subject
```

_output_
```bash
subject=CN = httpbin.org
```

Inspect SSL/TLS certificate information for debugging connection issues.

- openssl s_client connects and shows certificate
- openssl x509 displays certificate details
- Check dates for expiration

#### Mutual TLS (mTLS) configuration

```bash
# mTLS with client and CA certificates
curl --cert client-cert.pem \
  --key client-key.pem \
  --cacert ca-cert.pem \
  https://api.example.com/secure

# Using certificate bundle
curl --cert /etc/ssl/certs/client-cert.pem \
  --key /etc/ssl/private/client-key.pem \
  --cacert /etc/ssl/certs/ca-bundle.crt \
  https://secure-api.example.com

# Verify certificate against hostname
curl --cert-type PEM \
  --key-type PEM \
  --cacert ca.pem \
  https://api.example.com/data
```

_exec_
```bash
curl -i https://httpbin.org 2>&1 | grep -i "certificate\|subject"
```

_output_
```bash
* Certificate verified successfully
```

Configure mutual TLS authentication with client and server certificates.

- --cert client certificate file
- --key private key file
- --cacert CA certificate for verification

**Best practices:**

- Never use -k in production (disables security)
- Use proper certificates for production APIs
- Keep certificates and keys secure

**Common errors:**

- **certificate verify failed**: Use correct CA certificate or check certificate is valid

### Security Best Practices

Secure curl usage for APIs and services

**Keywords:** security, auth, sensitive, data

#### Protect sensitive data

```bash
# Store credentials in environment variables
API_KEY="$GITHUB_TOKEN"
curl -H "Authorization: token $API_KEY" https://api.github.com/user

# Use .netrc file for credentials (~/.netrc)
# Format: machine api.example.com login user password secret
# Permissions: chmod 600 ~/.netrc
curl --netrc https://api.example.com/data

# Read password securely without logging
read -sp "Enter API Key: " API_KEY
curl -H "X-API-Key: $API_KEY" https://api.example.com/data

# Don't include credentials in commands (use processes)
API_KEY=$(cat ~/.config/api-key)
curl -H "Authorization: Bearer $API_KEY" https://api.example.com
```

_exec_
```bash
echo 'Best practice: Use environment variables for secrets'
```

_output_
```bash
Best practice: Use environment variables for secrets
```

Safely handle sensitive credentials using environment variables and secure storage.

- Never hardcode credentials in scripts
- Use environment variables
- .netrc file for reusable credentials

#### Network security measures

```bash
# Use timeouts to prevent hanging requests
curl --connect-timeout 5 --max-time 30 https://api.example.com/data

# Limit supported protocols
curl --tlsv1.2 --tlsv1.3 https://api.example.com/data

# Disable insecure protocols
curl --disable https://api.example.com/data

# Use IPv4 or IPv6 only
curl -4 https://api.example.com  # IPv4 only
curl -6 https://api.example.com  # IPv6 only

# Restrict to specific IP ranges
curl --resolve api.example.com:443:192.0.2.1 https://api.example.com/data
```

_exec_
```bash
curl --connect-timeout 2 -m 5 https://httpbin.org/delay/1 2>&1 | head -3
```

_output_
```bash
{
  "delayed": true,
  "sleep": 1
```

Implement security best practices for network communication.

- --connect-timeout and --max-time prevent DoS
- Explicit TLS versions increase security
- IP restrictions prevent DNS spoofing

#### Logging and monitoring security

```bash
# Log requests without sensitive headers
curl -v https://api.example.com/data 2>&1 | sed 's/Authorization:.*/Authorization: [REDACTED]/'

# Monitor failed authentication attempts
log_file="/var/log/api-requests.log"
if ! curl -f -H "Authorization: Bearer $TOKEN" https://api.example.com/data >> "$log_file" 2>&1; then
  echo "Failed auth attempt at $(date)" >> "$log_file"
fi

# Audit API requests
{
  echo "Request to: https://api.example.com"
  echo "Time: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "Status: $(curl -s -o /dev/null -w '%{http_code}' https://api.example.com)"
} >> api-audit.log

# Redact sensitive data from responses
curl -s https://api.example.com/data | jq '.password = "[REDACTED]" | .token = "[REDACTED]"'
```

_exec_
```bash
echo "curl -v <url> 2>&1 | sed 's/Authorization:.*/Authorization: [REDACTED]/'"
```

_output_
```bash
curl -v <url> 2>&1 | sed 's/Authorization:.*/Authorization: [REDACTED]/'
```

Log and monitor API requests while protecting sensitive information.

- Redact credentials in logs
- Monitor for failed authentication
- Audit API usage for security analysis

**Best practices:**

- Never log full authentication headers
- Use HTTPS for all authenticated requests
- Rotate API keys and tokens regularly
- Monitor for unusual API access patterns

**Common errors:**

- **credentials exposed in bash history**: Use read -sp or environment variables

## Advanced Features

Advanced curl options and techniques

### Proxies & Redirects

Configure proxies and handle redirects

**Keywords:** proxy, redirect, forward, tunnel

#### Configure HTTP and SOCKS proxies

```bash
# HTTP proxy
curl -x http://proxy.example.com:8080 https://api.example.com/data

# SOCKS5 proxy
curl -x socks5://proxy.example.com:1080 https://api.example.com/data

# Proxy with authentication
curl -x http://user:pass@proxy.example.com:8080 https://api.example.com/data

# Use environment variable for proxy
export http_proxy="http://proxy.example.com:8080"
curl https://api.example.com/data
```

_exec_
```bash
curl -x "http://127.0.0.1:3128" https://httpbin.org/ip 2>&1 | head -5
```

_output_
```bash
* Connected to 127.0.0.1 (127.0.0.1) via SOCKS proxy
(Connection refused)
```

Route requests through HTTP or SOCKS proxies for privacy or network access.

- -x specifies proxy
- Format: protocol://[user:pass@]host:port
- Environment variables: http_proxy, https_proxy, no_proxy

#### Redirect handling

```bash
# Follow redirects
curl -L https://api.example.com/data

# Follow max 5 redirects
curl -L --max-redirs 5 https://api.example.com/data

# Don't follow redirects (default)
curl https://api.example.com/data

# Show redirect chain
curl -L -v https://example.com 2>&1 | grep -E "^> |< HTTP"

# POST data preserved across redirects (except method change)
curl -L -X POST -d "data=value" https://api.example.com/endpoint
```

_exec_
```bash
curl -L -s -w "\nFinal URL: %{url_effective}\n" https://httpbin.org/redirect/2 | tail -5
```

_output_
```bash
Final URL: https://httpbin.org/get
```

Control redirect behavior and trace redirect chains.

- -L enables automatic redirect following
- --max-redirs limits maximum redirects
- %{url_effective} shows final URL after redirects

#### Advanced redirect scenarios

```bash
# Preserve method on redirect
curl -L -X PUT -H "Content-Type: application/json" \
  -d '{"key":"value"}' https://api.example.com/endpoint

# Manual redirect handling
location=$(curl -s -I https://api.example.com/redirect | grep location | cut -d' ' -f2)
curl "$location"

# Redirect with cookie preservation
curl -b cookies.txt -L https://example.com/login
curl -b cookies.txt https://example.com/protected

# Debug redirects in detail
curl -L -v https://example.com 2>&1 | grep -E "^> GET|^< HTTP|^< location"
```

_exec_
```bash
curl -L -i https://httpbin.org/redirect-to?url=https://httpbin.org/get 2>&1 | grep -E "^HTTP|^location"
```

_output_
```bash
HTTP/1.1 302 FOUND
location: https://httpbin.org/get
HTTP/2 200
```

Implement complex redirect scenarios with method preservation and tracing.

- curl preserves POST/PUT on redirects only with -L
- Manual handling gives more control
- Trace redirects with -v flag

**Best practices:**

- Use -L for public APIs that redirect
- Limit redirects with --max-redirs to prevent loops
- Preserve POST data when necessary

**Common errors:**

- **maximum redirects exceeded**: Check for redirect loops or use --max-redirs

### Compression & Performance Optimization

Compression, connection reuse, and timing measurement

**Keywords:** compression, performance, gzip, deflate

#### Enable compression

```bash
# Request compression (ask server to compress response)
curl --compressed https://api.example.com/data

# Specify accepted compression methods
curl -H "Accept-Encoding: gzip, deflate, br" https://api.example.com/data

# Decompress and view
curl -s --compressed https://api.example.com/large-file | gzip -d

# Compare sizes
curl -s https://api.example.com/data | wc -c
curl -s --compressed https://api.example.com/data | wc -c
```

_exec_
```bash
curl -s --compressed https://httpbin.org/gzip | head -10
```

_output_
```bash
{
  "gzipped": true,
  "method": "GET",
  "origin": "192.0.2.1"
}
```

Enable compression for reduced bandwidth and faster transfers.

- --compressed requests gzip/deflate compression
- Automatic decompression by curl
- Savings are largest on big text responses

#### Connection pooling and optimization

```bash
# Reuse connection for multiple requests
url="https://api.example.com"
for i in {1..5}; do
  curl -s "$url/endpoint$i" &
done
wait

# Keep connection alive
curl -H "Connection: keep-alive" https://api.example.com/data

# Disable keep-alive if needed
curl -H "Connection: close" https://api.example.com/data

# Batch multiple requests efficiently
cat urls.txt | xargs -I {} curl -s {} > combined-output.txt
```

_exec_
```bash
echo "Connection optimization via HTTP/2, HTTP/1.1 default settings"
```

_output_
```bash
Connection optimization via HTTP/2, HTTP/1.1 default settings
```

Optimize connection handling for better performance.

- HTTP/2 multiplexes multiple requests
- Keep-alive reduces connection overhead
- Parallel requests with & and wait

#### Performance measurement and optimization

```bash
# Measure and compare performance
time curl -s https://api.example.com/data > /dev/null
time curl -s --compressed https://api.example.com/data > /dev/null

# Monitor bandwidth
curl -# https://api.example.com/large-file -o output

# Optimize with early termination
curl -s https://api.example.com/stream | head -100

# Measure request timing
curl -w "\nTime breakdown:\nDNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTransfer: %{time_starttransfer}s\nTotal: %{time_total}s\n" https://api.example.com/data
```

_exec_
```bash
curl -w "\n\nTotal: %{time_total}s\n" -o /dev/null -s https://httpbin.org/delay/1
```

_output_
```bash
Total: 1.234s
```

Measure and optimize request performance with timing analysis.

- Use time for overall performance
- -w provides detailed breakdown
- Compression cuts the bytes transferred

**Best practices:**

- Use --compressed for large responses
- Parallel requests with & for multiple APIs
- Monitor performance with -w timing data

**Common errors:**

- **server doesn't support compression**: Some servers do not support gzip, so handle an uncompressed response too