How to Use proxy with Python Requests

Key Takeaways

• A python requests proxy hides your real IP and distributes traffic across multiple addresses so target sites don’t flag your script.
• The requests library has built-in proxy support through a simple dictionary argument – no extra packages needed.
• Proxy authentication works by embedding your username and password inside the URL string.
• Sessions let you reuse one configuration across dozens of requests without repeating yourself.
• Environment variables like HTTP_PROXY and HTTPS_PROXY configure routing at the system level.
• Rotating proxies is essential for large scraping jobs – a single static IP gets blocked fast.
• Retry logic with HTTPAdapter prevents your pipeline from crashing when one connection goes down.
• Picking the right proxy type (residential, datacenter, ISP) matters more than pool size alone.

Why Use a Proxy with Python Requests?

Every request your script sends carries your IP address. The target server logs it. Send enough requests from that same address, and the server treats you like a bot – because technically you are one.

A proxy changes the equation. Your request goes to the proxy server first, which forwards it to the target using a different IP. The target sees the proxy address, not yours.

You can use proxies to send requests from IPs in specific countries, use proxy rotation to cycle through thousands of addresses, and keep your scraping infrastructure running even when IPs get burned.

Common Use Cases for Python Proxies

Web scraping is the obvious one. Price monitoring across Amazon, Walmart, or any e-commerce platform requires thousands of requests per day. Without a proxy, you get blocked after a few hundred.

SEO tools pull SERP data from Google across geolocations. Each request needs to look like it comes from a real user in that country.

Account management on platforms like Instagram or TikTok requires a separate IP per profile. Any time your script needs to look like traffic from somewhere else, use a proxy.

What Happens If You Send Requests Without a Proxy?

The first few requests work. Maybe the first few hundred. Then you start seeing 403 Forbidden responses. Or 429 Too Many Requests. The site serves you a CAPTCHA page instead of the data your request asked for.

Cloudflare and Akamai fingerprint your connection and block repeat offenders. Your IP gets blacklisted, and every future request from that address returns nothing useful.

What You Need Before Getting Started

Install the Requests Library

Install it with pip:

pip install requests

For project isolation, use a virtual environment: `python -m venv myenv && source myenv/bin/activate && pip install requests`. Confirm by running `import requests` inside a .py file. No errors means you are ready to use it.

Understand the Parts of a Proxy Address

The address follows this format:

protocol://username:password@ip:port

Use `http`, `https`, or `socks5` as the scheme. Example: `http://user123:pass456@185.199.42.11:8080`. The port is not optional – skip it and the request fails.

Choose the Right Proxy Provider

Free lists are tempting. They are also unreliable, slow, and often compromised. For anything beyond a quick test, use a paid provider.

What to look for: pool size, geographic coverage, and protocol support (HTTP and SOCKS5). Residential bandwidth costs $1.00 to $15.00 per GB. Datacenter IPs run $0.50 to $2.00 per month.

Best Proxy Solution for Python Requests

Why ProxyWing Is a Strong Choice for Python Requests

Proxywing runs a pool of over 70 million IPs across 190+ countries. Residential bandwidth starts at $1.00/GB. Datacenter options cost $0.90/month per IP. Both support HTTP and SOCKS5, and sticky sessions hold for up to 7 days.

If your script needs to use a proxy session on a target site – say, staying logged in while scraping paginated results – a 7-day sticky session keeps your IP consistent without manual rotation logic. ProxyWing integrates with Afina, AdsPower and GoLogin, which helps if your python requests proxy setup feeds into a multi-account operation. Uptime sits at 99%.

Which Proxy Type to Choose for Your Task

Residential proxies come from real household devices. Use them for targets with aggressive anti-bot systems – Google, Amazon, social platforms.

Datacenter proxies are faster and cheaper. Use them for smaller e-commerce sites and public APIs.

ISP proxies sit in between. Use them for sustained sessions on moderately protected sites.

How to Use a Proxy with Python Requests

Step 1: Import Python Requests

Create a new .py file and start with the import:

import requests

That gives you access to `requests.get()`, `requests.post()`, and the Session class. No other package is needed to use proxies with basic requests.

Step 2: Set Proxies Directly in Requests

Python requests accepts a `proxies` argument – a dictionary that maps each protocol to a proxy address:

proxies = {

    "http": "http://185.199.42.11:8080",

    "https": "http://185.199.42.11:8080"

}

response = requests.get("https://httpbin.org/ip", proxies=proxies)

print(response.json())

Both `http` and `https` keys should point to your server. If you only set one, requests on the other protocol bypass it. Always use both.

Step 3: Add Proxy Authentication

Most proxy providers require a username and password. Embed them in the URL:

proxies = {

    "http": "http://user123:pass456@gate.proxywing.com:7000",

    "https": "http://user123:pass456@gate.proxywing.com:7000"

}

response = requests.get("https://httpbin.org/ip", proxies=proxies)

print(response.status_code)

The credentials go before the `@` symbol. The requests library parses them and sends the authorization header. If your password contains special characters, use `urllib.parse.quote()`.

Step 4: Send a Test Request Through the Proxy

Before running your full script, verify the connection works:

import requests

proxies = {

    "http": "http://user123:pass456@gate.proxywing.com:7000",

    "https": "http://user123:pass456@gate.proxywing.com:7000"

}

try:

    response = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=10)

    print(f"Proxy IP: {response.json()['origin']}")

except requests.exceptions.ProxyError:

    print("Connection failed")

except requests.exceptions.Timeout:

    print("Request timed out")

The `httpbin.org/ip` endpoint returns the IP the server sees. If the result shows a proxy IP instead of yours, it works. Always use a `timeout` so your request does not hang.

Step 5: Use a Session for Repeated Requests

Creating a new connection for every request is wasteful. Use a Session object in python requests – it reuses the TCP link and keeps your config in one place:

import requests

session = requests.Session()

session.proxies = {

    "http": "http://user123:pass456@gate.proxywing.com:7000",

    "https": "http://user123:pass456@gate.proxywing.com:7000"

}

for url in ["https://httpbin.org/ip", "https://httpbin.org/headers"]:

    response = session.get(url, timeout=10)

    print(f"{url}: {response.status_code}")

Set the proxy once on the session. Every request through that session routes through it automatically. Sessions also handle cookies, which matters when the target tracks request state across requests.

How to Set Proxies via Environment Variables

When Environment Variables Make Sense

Hardcoding addresses inside your .py file works for testing. For production, use environment variables. They keep credentials out of source code. Python requests picks these up automatically.

Example of Using HTTP_PROXY and HTTPS_PROXY

Set them in your terminal before you run the script:

export HTTP_PROXY="http://user123:pass456@gate.proxywing.com:7000"

export HTTPS_PROXY="http://user123:pass456@gate.proxywing.com:7000"

Then your code needs no proxy configuration:

import requests

response = requests.get("https://httpbin.org/ip")

print(response.json())

The requests library checks for these variables on every request. To bypass the proxy for one call, pass `proxies={}`.

How to Rotate Proxies in Python Requests

Build a List of Proxies

Rotation starts with having multiple addresses. Store them in a list:

proxy_list = [

    "http://user:pass@proxy1.example.com:8000",

    "http://user:pass@proxy2.example.com:8000",

    "http://user:pass@proxy3.example.com:8000",

]

For larger pools, load from a file – one address per line.

Validate Proxies Before Using Them

Dead entries slow everything down. Test each one before you use it:

import requests

def check(proxy_url):

    try:

        r = requests.get("https://httpbin.org/ip",

            proxies={"http": proxy_url, "https": proxy_url}, timeout=5)

        return r.status_code == 200

    except Exception:

        return False

active = [p for p in proxy_list if check(p)]

Run this check before every scraping session. What worked yesterday might be dead today.

Rotate Proxies with a Simple Custom Method

The simplest rotation picks a random entry for each request:

import random

import requests

def get_rotated(url, pool):

    pick = random.choice(pool)

    return requests.get(url,

        proxies={"http": pick, "https": pick}, timeout=10)

Random selection works for small jobs. For anything over a few thousand requests, round-robin gives better distribution.

Use Sessions or Logic to Retry with a New Proxy

When one fails mid-job, your script should use the next entry:

import random

import requests

def fetch_with_retry(url, pool, max_retries=3):

    for attempt in range(max_retries):

        pick = random.choice(pool)

        try:

            resp = requests.get(url,

                proxies={"http": pick, "https": pick}, timeout=10)

            if resp.status_code == 200:

                return resp

        except requests.exceptions.RequestException:

            continue

    return None

Each attempt picks a different address. After three tries, the function returns `None` so your loop can log the failed request.

Advanced Proxy Rotation for Larger Tasks

At scale, random selection is not enough. Track which IPs perform well and score them. For projects above 100,000 requests per day, use a provider with built-in rotating gateways. ProxyWing’s residential endpoint handles rotation server-side.

Common Errors When Using Proxies with Python Requests

Proxy Authentication Errors

A 407 response means your proxy credentials are wrong. Double-check the username and password in the proxy URL. Common mistakes: a typo, use of the wrong port, or forgetting to URL-encode special characters.

Connection Timeouts and Network Failures

ConnectTimeout usually means the proxy server is down. Use a different one from your list. If all of them time out, the problem might be on your end – a firewall blocking outbound connections. Always set explicit timeouts: `timeout=(5, 15)` gives 5 seconds for the connection and 15 for the response.

HTTP 403, 407, and 429 Errors

403 Forbidden – the target blocked your request. Use a residential proxy or add headers that mimic a real browser to the request.

429 Too Many Requests – you are sending requests too fast. Add delays between requests and use more proxy IPs to spread the request load.

SSL and HTTPS Issues

SSLError happens when the proxy intercepts HTTPS traffic with its own certificate. You can use `verify=False`, but that disables security checks. Better to get the proxy provider’s CA certificate and pass it with `verify=”/path/to/ca.crt”`.

How to Retry Failed Requests

Common Causes of Failed Requests

Requests fail for three main reasons: the proxy is dead, the target blocked the proxy IP, or the network dropped the request. Remove dead proxies from rotation. Use a fresh proxy IP for blocked ones. Retry dropped requests.

Add Retry Logic with Requests and HTTPAdapter

The python requests library supports automatic retries through `urllib3.util.retry.Retry` and `HTTPAdapter`:

import requests

from requests.adapters import HTTPAdapter

from urllib3.util.retry import Retry

session = requests.Session()

retry_strategy = Retry(total=3, backoff_factor=1,

    status_forcelist=[429, 500, 502, 503, 504])

adapter = HTTPAdapter(max_retries=retry_strategy)

session.mount("http://", adapter)

session.mount("https://", adapter)

The `backoff_factor=1` adds increasing delays: 1 second, then 2, then 4. The `status_forcelist` tells the adapter which codes trigger a retry.

Build a Simple Custom Retry Wrapper

For more control, use a custom wrapper to pick a new proxy on each retry:

import time, random, requests

def resilient_request(url, pool, retries=3, delay=2):

    for i in range(retries):

        pick = random.choice(pool)

        try:

            resp = requests.get(url,

                proxies={"http": pick, "https": pick}, timeout=10)

            if resp.status_code in [200, 301, 302]:

                return resp

            if resp.status_code == 429:

                time.sleep(delay * (i + 1))

        except requests.exceptions.RequestException:

            time.sleep(delay)

    return None

How to Reduce the Chance of Error 429

Rate limiting is the most common issue. First, use more proxy IPs to spread requests across addresses. Second, use randomized delays between requests – `time.sleep(random.uniform(1.0, 3.5))`. Third, use realistic request headers including a `User-Agent` that matches a real browser. Fourth, respect the `robots.txt` crawl-delay directive.

Best Practices for Using Proxies with Python Requests

Use High-Quality Proxies

Free proxy lists share their IPs with hundreds of other users. Those IPs are already flagged. A proxy that costs nothing saves you nothing when every request returns a 403. Use a provider with a large, clean proxy pool.

Match Proxy Type to the Target Site

Google, Amazon, and Facebook invest heavily in bot detection. Datacenter proxy IPs get caught fast. Use residential proxies for hard targets. Smaller sites work fine with datacenter proxies – no reason to use $1.00/GB residential when $0.90/month datacenter proxies do the job.

Avoid Overloading a Single IP

Sending 100 requests per second from one proxy address looks automated. Distribute your requests across the proxy pool. If you use 20 proxy IPs, send no more than 5 requests per minute from each.

Monitor Response Codes and Proxy Health

Log every response code. A sudden spike in 403 or 429 responses means the target updated its detection, or your proxy IPs are getting stale. Drop any proxy that falls below 90% success rate.

Conclusion

Setting up a python requests proxy is straightforward. The requests library handles proxy routing, authentication, and session management with minimal code. The harder part is everything around it – picking the right proxy type, rotating proxy IPs, and handling failed requests gracefully.

Start with the basics. Get a single request working through a proxy. Then add sessions, proxy rotation, and retry logic piece by piece. The foundation matters, and once your python requests proxy setup is solid, scaling from 100 requests to 100,000 is mostly about adding more proxy IPs.