|

How to Use a Proxy in Selenium v4 with Firefox & Python (2024)

ByAbdelhadi Dyouri Updated on

Have you been struggling to get a proxy to work in Selenium while using the Firefox web driver?

Rest assured, in this tutorial, I will give you Python Selenium code that actually works for proxying requests in Selenium with a Firefox web driver.

Selenium is a browser automation tool that is used to build bots, test web applications and scrape data from the internet. Its Python library is a popular choice for web automation and web scraping.

If you’re not sure what’s a proxy, check out my What is a Web Scraping Proxy? guide.

Without further ado, here is a clear, step-by-step guide on how to use a proxy in Selenium v4 with Firefox in Python.

How to use a proxy in selenium v4 with Firefox

Note: If you are using the Chrome webdriver, checkout my How to Use a Proxy in Selenium v4 with Chrome tutorial.

How to Use a Proxy in Selenium v4 with Firefox

For experienced developers with no time for explanations, here is the Python code you need to use a Proxy in Selenium with the Firefox driver:

import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "207.2.120.19")
options.set_preference("network.proxy.http_port", 80)

driver = webdriver.Firefox(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

The output should have your proxy server’s IP address instead of your local machine’s IP:

selenium proxy in firefox

That’s it! You are now using a proxy in Selenium with a Firefox web driver. If the code above is unclear, or you can’t get it to work, then carefully read the full detailed guide below.

Step 01 – Installing Selenium

First, make sure you have Firefox installed on your system.

Next, create a new Python folder called selenium-proxy:

mkdir selenium-proxy

Create a new Python virtual environment inside it:

python -m venv env

Activate the virtual environment:

source env/bin/activate

Install Selenium:

pip install selenium

Next, add a file called scraper.py and then add the following code to it:

from selenium import webdriver

driver = webdriver.Firefox()
driver.get("https://www.python.org")

print(driver.title)

print(driver.current_url)

driver.close()

Here’s a brief explanation of each part:

  1. from selenium import webdriver: This line imports the Selenium module which provides tools for automating web browsers.
  2. driver = webdriver.Firefox(): This line creates an instance of the Firefox web driver, which allows Python to control the Firefox web browser.
  3. driver.get("https://www.python.org"): This instructs the web driver to navigate to the URL “https://www.python.org”.
  4. print(driver.title): This prints the title of the web page currently loaded in the browser (in this case, the Python website).
  5. print(driver.current_url): This prints the current URL of the web page loaded in the browser.
  6. driver.close(): This closes the browser window opened by the web driver.

Now run scraper.py:

python scraper.py

Output

Welcome to Python.org
https://www.python.org/

The preceding output indicates your Selenium Firefox environment is working properly. If you are facing issues, check that you’ve correctly set up your environment.

Step 02 – Using a Proxy with Selenium in Firefox

With Selenium and Firefox successfully set up. Let’s add a proxy.

Inside your selenium-proxy project folder, create a new file called proxy.py, and add the following code to it:

import time
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.by import By

options = Options()
options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.http", "207.2.120.19")
options.set_preference("network.proxy.http_port", 80)

driver = webdriver.Firefox(
    options=options
)

driver.get("http://httpbin.org/ip")

print(driver.find_element(By.TAG_NAME, "body").text)

time.sleep(5)
driver.close()

Here is a breakdown of the preceding code:

  1. import time: This imports the time module, which is used to add time delays in the script.
  2. from selenium import webdriver: This imports the webdriver module from the Selenium library, which provides tools for automating web browsers.
  3. from selenium.webdriver.firefox.options import Options: This imports the Options class from the Selenium Firefox module, which allows customizing browser options.
  4. from selenium.webdriver.common.by import By: This imports the By class from the Selenium library, which provides mechanisms for selecting elements in the web page.
  5. options = Options(): This creates an instance of the Options class, which is used to set browser preferences.
  6. options.set_preference("network.proxy.type", 1): This sets the browser’s network proxy type to use a manual proxy configuration.
  7. options.set_preference("network.proxy.http", "207.2.120.19"): This sets the HTTP proxy address to “207.2.120.19”.
  8. options.set_preference("network.proxy.http_port", 80): This sets the HTTP proxy port to 80.
  9. driver = webdriver.Firefox(options=options): This initializes a new instance of the Firefox web driver with the custom options set.
  10. driver.get("http://httpbin.org/ip"): This instructs the browser to navigate to the URL “http://httpbin.org/ip”, which is a website that returns information about the client’s IP address.
  11. print(driver.find_element(By.TAG_NAME, "body").text): This finds the body element of the web page and prints its text content, which in this case should contain information about the client’s IP address.
  12. time.sleep(5): This pauses the script execution for 5 seconds, allowing time for the web page to load and display its content.
  13. driver.close(): This closes the browser window opened by the web driver.

The output should be as follows:

{
  "origin": "207.2.120.19"
}

To use an SSL Proxy with Selenium in Firefox, implement the following options:

options.set_preference("network.proxy.type", 1)
options.set_preference("network.proxy.ssl", SSL_IP)
options.set_preference("network.proxy.ssl_port", SSL_PORT)

To use proxy authentication in Firefox, check out the next step.

Step 03 – Selenium Proxy Authentication with Python in Firefox

How to use Proxy Authentication in Selenium v4 with Firefox

If you want to use a proxy that requires authentication with a username and password. The simplest way to achieve this is by using the selenium-wire extension. Selenium Wire makes it really easy to use both unauthenticated and authenticated proxies in Selenium.

To install Selenium Wire:

pip install selenium-wire

To use proxy authentication with Selenium Wire in Firefox, you can just pass the username and password of your proxy server in the options dictionary like so:

import time
from seleniumwire import webdriver

options = {
'proxy': {
    'http': 'http://USERNAME:PASSWORD@PROXY_SERVER:PORT',
    'https': 'https://USERNAME:PASSWORD@PROXY_SERVER:PORT',
    'no_proxy': 'localhost,127.0.0.1,dev_server:8080'
    }
}

driver = webdriver.Firefox(seleniumwire_options=options)

driver.get('http://httpbin.org/ip')
time.sleep(5)
driver.close()

If you’re proxy uses the Bearer scheme, you can pass your authorization token in a Proxy-Authorization header using the custom_authorization option:

options = {
    'proxy': {
        'https': 'https://192.168.10.100:8888',  # No username or password used
        'custom_authorization': 'Bearer mytoken123'  # Custom Proxy-Authorization header value
    }
}

For more on how to use proxies, check out the Selenium Wire Documentation

Sources

Conclusion

Congrats, you now know how to use a proxy in Python Selenium v4.0 with a Firefox web driver. You also learned how to use proxy authentication in Selenium for proxy servers protected with a username and password.

FAQ: Selenium, Firefox, and Proxies

Why is using proxies important in web scraping with Selenium and Firefox?

Proxies are essential in web scraping to mask your real IP address and avoid being blocked or rate-limited by websites.

By rotating through a pool of proxies, you can distribute requests across multiple IP addresses, mitigating the risk of detection and ensuring uninterrupted scraping operations.

Can I rotate proxies automatically while scraping with Selenium and Firefox?

Yes, you can rotate proxies automatically by managing a pool of proxies and switching between them in your Selenium script.

Implementing a proxy rotation mechanism involves periodically selecting a new proxy from the pool and updating the Firefox options accordingly. This approach helps distribute requests evenly and reduces the likelihood of IP bans or detection.

Are there any limitations or considerations when using proxies with Selenium and Firefox for web scraping?

While proxies provide anonymity and flexibility in web scraping, there are some considerations to keep in mind.

Firstly, free proxies may be unreliable or slow, affecting scraping performance. Additionally, websites may detect and block proxy IP addresses, requiring frequent updates to your proxy pool. Also, using proxies may incur additional costs, especially when utilizing premium or rotating proxy services.

Similar Posts