Rest assured, in this tutorial, I will give you Python Selenium code that actually works for proxying requests in Selenium with a Firefox web driver.

Selenium is a browser automation tool that is used to build bots, test web applications and scrape data from the internet. Its Python library is a popular choice for web automation and web scraping.

If you’re not sure what’s a proxy, check out my What is a Web Scraping Proxy? guide.

Without further ado, here is a clear, step-by-step guide on how to use a proxy in Selenium v4 with Firefox in Python.

Note: If you are using the Chrome webdriver, checkout my How to Use a Proxy in Selenium v4 with Chrome tutorial.

How to Use a Proxy in Selenium v4 with Firefox

For experienced developers with no time for explanations, here is the Python code you need to use a Proxy in Selenium with the Firefox driver:

import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "207.2.120.19")
options.set_preference("network.proxy.http_port", 80)

driver = webdriver.Firefox(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

The output should have your proxy server’s IP address instead of your local machine’s IP:

That’s it! You are now using a proxy in Selenium with a Firefox web driver. If the code above is unclear, or you can’t get it to work, then carefully read the full detailed guide below.

Step 01 – Installing Selenium

First, make sure you have Firefox installed on your system.

Next, create a new Python folder called selenium-proxy:

mkdir selenium-proxy

Create a new Python virtual environment inside it:

python -m venv env

Activate the virtual environment:

source env/bin/activate

Install Selenium:

pip install selenium

Next, add a file called scraper.py and then add the following code to it:

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.python.org")

print(driver.title)

print(driver.current_url)

driver.close()

Here’s a brief explanation of each part:

1. from selenium import webdriver: This line imports the Selenium module which provides tools for automating web browsers.
2. driver = webdriver.Firefox(): This line creates an instance of the Firefox web driver, which allows Python to control the Firefox web browser.
3. driver.get("https://www.python.org"): This instructs the web driver to navigate to the URL “https://www.python.org”.
4. print(driver.title): This prints the title of the web page currently loaded in the browser (in this case, the Python website).
5. print(driver.current_url): This prints the current URL of the web page loaded in the browser.
6. driver.close(): This closes the browser window opened by the web driver.

Now run scraper.py:

python scraper.py

Output

Welcome to Python.org
https://www.python.org/

The preceding output indicates your Selenium Firefox environment is working properly. If you are facing issues, check that you’ve correctly set up your environment.

Step 02 – Using a Proxy with Selenium in Firefox

With Selenium and Firefox successfully set up. Let’s add a proxy.

Inside your selenium-proxy project folder, create a new file called proxy.py, and add the following code to it:

import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "207.2.120.19")
options.set_preference("network.proxy.http_port", 80)

driver = webdriver.Firefox(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

Here is a breakdown of the preceding code:

1. import time: This imports the time module, which is used to add time delays in the script.
2. from selenium import webdriver: This imports the webdriver module from the Selenium library, which provides tools for automating web browsers.
3. from selenium.webdriver.firefox.options import Options: This imports the Options class from the Selenium Firefox module, which allows customizing browser options.
4. from selenium.webdriver.common.by import By: This imports the By class from the Selenium library, which provides mechanisms for selecting elements in the web page.
5. options = Options(): This creates an instance of the Options class, which is used to set browser preferences.
6. options.set_preference("network.proxy.type", 1): This sets the browser’s network proxy type to use a manual proxy configuration.
7. options.set_preference("network.proxy.http", "207.2.120.19"): This sets the HTTP proxy address to “207.2.120.19”.
8. options.set_preference("network.proxy.http_port", 80): This sets the HTTP proxy port to 80.
9. driver = webdriver.Firefox(options=options): This initializes a new instance of the Firefox web driver with the custom options set.
10. driver.get("http://httpbin.org/ip"): This instructs the browser to navigate to the URL “http://httpbin.org/ip”, which is a website that returns information about the client’s IP address.
11. print(driver.find_element(By.TAG_NAME, "body").text): This finds the body element of the web page and prints its text content, which in this case should contain information about the client’s IP address.
12. time.sleep(5): This pauses the script execution for 5 seconds, allowing time for the web page to load and display its content.
13. driver.close(): This closes the browser window opened by the web driver.

The output should be as follows:

{
  "origin": "207.2.120.19"
}

To use an SSL Proxy with Selenium in Firefox, implement the following options:

options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.ssl", SSL_IP)
options.set_preference("network.proxy.ssl_port", SSL_PORT)

To use proxy authentication in Firefox, check out the next step.

Step 03 – Selenium Proxy Authentication with Python in Firefox

If you want to use a proxy that requires authentication with a username and password. The simplest way to achieve this is by using the selenium-wire extension. Selenium Wire makes it really easy to use both unauthenticated and authenticated proxies in Selenium.

To install Selenium Wire:

pip install selenium-wire

To use proxy authentication with Selenium Wire in Firefox, you can just pass the username and password of your proxy server in the options dictionary like so:

import time
from seleniumwire import webdriver

options = {
'proxy': {
    'http': 'http://USERNAME:PASSWORD@PROXY_SERVER:PORT',
    'https': 'https://USERNAME:PASSWORD@PROXY_SERVER:PORT',
    'no_proxy': 'localhost,127.0.0.1,dev_server:8080'
    }
}

driver = webdriver.Firefox(seleniumwire_options=options)

driver.get('http://httpbin.org/ip')
time.sleep(5)
driver.close()

If you’re proxy uses the Bearer scheme, you can pass your authorization token in a Proxy-Authorization header using the custom_authorization option:

options = {
    'proxy': {
        'https': 'https://192.168.10.100:8888',  # No username or password used
        'custom_authorization': 'Bearer mytoken123'  # Custom Proxy-Authorization header value
    }
}

For more on how to use proxies, check out the Selenium Wire Documentation

Sources

• Python Selenium Documentation
• Selenium Wire Documentation

Conclusion

Congrats, you now know how to use a proxy in Python Selenium v4.0 with a Firefox web driver. You also learned how to use proxy authentication in Selenium for proxy servers protected with a username and password.

FAQ: Selenium, Firefox, and Proxies

Why is using proxies important in web scraping with Selenium and Firefox?

Proxies are essential in web scraping to mask your real IP address and avoid being blocked or rate-limited by websites.

By rotating through a pool of proxies, you can distribute requests across multiple IP addresses, mitigating the risk of detection and ensuring uninterrupted scraping operations.

Can I rotate proxies automatically while scraping with Selenium and Firefox?

Yes, you can rotate proxies automatically by managing a pool of proxies and switching between them in your Selenium script.

Implementing a proxy rotation mechanism involves periodically selecting a new proxy from the pool and updating the Firefox options accordingly. This approach helps distribute requests evenly and reduces the likelihood of IP bans or detection.

Are there any limitations or considerations when using proxies with Selenium and Firefox for web scraping?

While proxies provide anonymity and flexibility in web scraping, there are some considerations to keep in mind.

Firstly, free proxies may be unreliable or slow, affecting scraping performance. Additionally, websites may detect and block proxy IP addresses, requiring frequent updates to your proxy pool. Also, using proxies may incur additional costs, especially when utilizing premium or rotating proxy services.

I’ve spent hours figuring out how to use a proxy in Selenium with a Chrome driver for one of my web scraping projects.

Google search results were full of outdated tutorials with code that doesn’t work. So, I went into the depths of the latest Selenium documentation, and I finally figured it out. Now, I present to you a Selenium proxy tutorial with Python code that actually works.

Notes

• I am using the Chrome driver in this tutorial.

• If you’re not sure what’s a proxy, check out my What is a Web Scraping Proxy? guide.

Selenium is a browser automation tool that is used to build bots, test web applications and scrape data from the internet. Its Python library is a popular choice for web automation and web scraping.

Note: If you are using the Firefox Webdriver, checkout my How to Use a Proxy in Selenium v4 with Firefox & Python tutorial.

How to Use a Proxy in Selenium

For experienced developers, here is a quick and short answer for how to use a Proxy in Selenium with the chrome driver:

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

PROXY = "http://207.2.120.19:80"

options = Options()
options.add_argument(f'--proxy-server={PROXY}')


driver = webdriver.Chrome(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

The output should be as follows:

That’s it, you are now using a proxy with selenium. If the preceding code is unclear, or you can’t get it to work, then carefully read the full detailed guide below.

Step 01 – Getting Selenium to Work

Previously, you had to download a chromedriver web driver for Selenium to work properly. With Selenium version 4, you no longer need to install a web driver.

However, if Selenium is not working for you, you may still need to download the chromedriver. To do so, follow these steps:

• Go to Chrome’s Downloads Page
• Select your Chrome’s version.
• If you are using Chrome version 115 or newer, please consult the Chrome for Testing availability dashboard.
• Select the chromedriver that’s suitable for your OS from the table, download it, then add it to your PATH.

Make sure to run the chromedriver and your Chrome or Chromium browser at least once before proceeding. This step may not be necessary for some users:

$ chromedriver

Now, create a new Python folder called selenium-proxy:

mkdir selenium-proxy

Create a new Python virtual environment inside it:

python -m venv env

Activate the virtual environment:

source env/bin/activate

Install selenium:

pip install selenium

Next, add a file called scraper.py and then add the following code to it:

# scraper.py
import time
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.python.org")

print(driver.title)

print(driver.current_url)

time.sleep(5)
driver.close()

The preceding script opens the Python website using a Chrome WebDriver, prints the title and URL of the page, pauses for 5 seconds, and then closes the browser.

Here is a more detailed overview:

1. Import Libraries:
   • time: Used for adding delays in the script.
   • webdriver from the selenium library: Provides a common interface for interacting with web browsers. In this case, it’s using the Chrome WebDriver.
2. Instantiate WebDriver:
   • driver = webdriver.Chrome(): This line creates an instance of the Chrome WebDriver, allowing the script to control and automate the Google Chrome browser.
3. Navigate to a URL:
   • driver.get("https://www.python.org"): Instructs the WebDriver to open the specified URL, in this case, the official Python website.
4. Print Page Title and URL:
   • print(driver.title): Outputs the title of the current web page opened by the WebDriver (in this case, the Python website).
   • print(driver.current_url): Outputs the URL of the current web page.
5. Pause Execution (Sleep):
   • time.sleep(5): Introduces a 5-second delay in the script’s execution. This allows you to see the website loaded on chrome, instead of being closed right away.
6. Close the Browser:
   • driver.close(): Closes the browser window that the WebDriver is controlling.

Now, run scraper.py:

python scraper.py

You should see the following output:

Welcome to Python.org
https://www.python.org/

This means Selenium and chromedriver are working properly.

Step 02 – Using a Proxy with Selenium

Now, let’s move on to how to use a proxy in your python selenium code.

Inside your selenium-proxy project folder, create a new file called proxy.py, and add the following code to it:

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

PROXY = "http://207.2.120.19:80"

options = Options()
options.add_argument(f'--proxy-server={PROXY}')


driver = webdriver.Chrome(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

Let’s break down the code step by step:

1. Import Libraries:

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

Here are the imports:

• time: Provides time-related functions for introducing delays.
• webdriver: From the Selenium library, used for automating web browser interactions.
• Options: A class from selenium.webdriver.chrome.options to configure Chrome WebDriver options.
• By: A class from selenium.webdriver.common.by to specify the mechanism used to locate elements on a webpage.

2. Define Proxy Information:

PROXY = "http://207.2.120.19:80"

Sets a proxy server address (http://207.2.120.19:80). This is used to route the web traffic through a specified proxy server.

3. Configure Chrome WebDriver with Proxy:

options = Options()
options.add_argument(f'--proxy-server={PROXY}')

• Creates an instance of Options class to configure the Chrome WebDriver.
• Adds a command-line argument to specify the proxy server for the Chrome WebDriver.

4. Initialize WebDriver:

driver = webdriver.Chrome(options=options)

Creates a Chrome WebDriver instance with the configured options.

5. Navigate to a URL:

driver.get("http://httpbin.org/ip")

Directs the WebDriver to open the specified URL (http://httpbin.org/ip).

6. Find and Print Page Content:

print(driver.find_element(By.TAG_NAME, "body").text)

Finds the <body> element of the webpage and prints its text content. By.TAG_NAME is used to locate the element by its HTML tag name.

7. Pause Execution (Sleep):

time.sleep(5)

Introduces a 5-second delay so that the Chrome browser doesn’t close right away.

8. Close the Browser:

driver.close()

Closes the browser window controlled by the WebDriver.

Output

The output should have your proxy’s IP address instead of your local machine’s IP address:

{
  "origin": "207.2.120.19"
}

Step 03 – Selenium Proxy Authentication with Python in Chrome

If you want to use a proxy that requires authentication with a username and password. The simplest way to achieve this is by using the selenium-wire extension.

Selenium Wire extends Selenium’s Python bindings to give you access to the underlying requests made by the browser. You code in the same way as you do with Selenium, but you get extra APIs for inspecting requests and responses. Selenium Wire also makes it really easy to use both unauthenticated and authenticated proxies in Selenium.

To install Selenium Wire:

pip install selenium-wire

Here is how to use an open unauthenticated proxy with Selenium Wire:

import time
from seleniumwire import webdriver

PROXY = "45.77.194.89:38869"

options = {
    'proxy': {
        'http': f'http://{PROXY}',
        'https': f'https://{PROXY}',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)

driver.get('http://httpbin.org/ip')
time.sleep(5)
driver.close()

As you can see, you can just specify the IP address and port, pass the options to a seleniumwire_options parameter, and that’s it!

To use proxy authentication with Selenium Wire, you can just pass the username and password of your proxy server in the options dictionary like so:

import time
from seleniumwire import webdriver

options = {
    'proxy': {
        'http': 'http://USERNAME:PASSWORD@PROXY_SERVER:PORT',
        'https': 'https://USERNAME:PASSWORD@PROXY_SERVER:PORT',
        'no_proxy': 'localhost,127.0.0.1'
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)

driver.get('http://httpbin.org/ip')
time.sleep(5)
driver.close()

If your proxy uses the Bearer scheme, you can pass your authorization token in a Proxy-Authorization header using the custom_authorization option:

options = {
    'proxy': {
        'https': 'https://192.168.10.100:8888',  # No username or password used
        'custom_authorization': 'Bearer mytoken123'  # Custom Proxy-Authorization header value
    }
}

Conclusion

Congrats, you now know how to use a proxy in Python Selenium v4.0 with a Chrome web driver. You also learned how to use proxy authentication in Selenium for proxy servers protected with a username and password.

Sources

• Python Selenium Documentation
• Selenium Wire Documentation

FAQs

Why use a proxy with Selenium?

Using a proxy with Selenium offers several benefits:

1. IP Anonymity: Proxies allow you to conceal your IP address, providing anonymity during web scraping or automated testing.
2. Geographic Targeting: Proxies enable you to access websites as if you were located in a different geographic region, useful for testing or gathering region-specific data.
3. Bypass Restrictions: Proxies help bypass IP-based restrictions, allowing access to websites that may block requests from specific IPs.
4. Load Management: Distributing requests through different IP addresses using proxies can help manage server loads and reduce the risk of being flagged for excessive traffic.
5. Content Verification: Proxies aid in verifying content availability and consistency across different regions or networks, ensuring a more comprehensive testing approach.

Can using a proxy with Selenium help prevent IP blocking during web scraping?

Yes, using a proxy with Selenium can help prevent IP blocking during web scraping.

By rotating IP addresses through proxies, you distribute requests, making it harder for websites to detect and block automated traffic.

This enhances the resilience of your web scraping activities and reduces the risk of being flagged as a bot.

Are there free proxies available for use with Selenium, or do I need to purchase them?

Both free and paid proxies are available. Free proxies may have limitations in terms of reliability, speed, and security.

For more demanding tasks or projects, it’s often advisable to consider paid proxy services that offer better performance, dedicated support, and enhanced features.