eBay Web Scraping Guide: Build Powerful Scrapers with Puppeteer, Playwright & Python

By using an eBay scraper, it is much easier and faster to collect this information by copying the data manually. In this comprehensive guide, we will discuss web scraping and how you can build powerful eBay scrapers using tools like Puppeteer, Playwright, and other Python tools. Let’s start off with a summary of the key points in this guide. 

Key Takeaways

• Web scraping allows businesses to collect public data like listings, prices, variants, seller details, and sales signals. 
• Choosing the right scaling tool depends on target page complexity, scale, and any other specific needs. Puppeteer, Playwright, and Python-based tools are the three popular choices. 
• Understanding eBay’s page structure is critical for reliable web scraping.
• Scaling e-commerce scraping requires anti-block strategies such as integrating proxies, creating delays between requests, and using proper headers. 
• Use BrowserQL for high-volume, enterprise-level web scraping.

What Is eBay Web Scraping?

It is the process of automatically collecting publicly available data from eBay. This process is executed by scripts or software tools, instead of gathering the information manually. By using eBay scrapers, it is easier and faster to collect loads of data that can then be fed into analytics tools to help guide decisions.

Depending on your goal, web scraping usually includes collecting a wide range of data. Some of the information that is usually extracted from product listing pages, search results, and seller profiles includes: 

• Product listings: This can include titles, descriptions, categories, and images.
• Prices: Product’s current prices, discounts, bids, and Buy It Now values.
• Product variants: These can include size, color, condition, and availability.
• Seller Information: This can include seller names, ratings, feedback counts, and locations.
• Sales signals: Includes quantity available and number of items sold

When web scraping, it is upon you to determine what data you want your scraper to collect. To avoid wasting time and resources, it is always best to design the eBay scrapers to only collect the data you need. 

Why Scrape eBay?

• Price monitoring: This data collection process helps businesses track competitor pricing in real time, which can help guide their pricing strategies to stay competitive.
• Competitor research:Web scraping also helps businesses analyze competitor listings, seller performance, and pricing strategies.
• Product and market analysis: Before getting into a new market, businesses need to identify high-demand products, popular variants, and emerging trends. This is effectively done through web scraping. 
• SEO and listing optimization research: Businesses use scraped data to learn which keywords and listing formats perform best to improve visibility.
• Web automations and efficiency: Web scraping also helps automate repetitive research tasks and scale data collection for reports and alerts.

Understanding eBay’s Page & DOM Structure

Before getting started with the web scraping process, it is important to understand how eBay’s web pages are structured and what elements your script will target. Let’s walk you through an overview of this platform’s page and DOM structure:

• Search result pages: Each product appears as a repeated block (listing card) and usually contains the title, price, shipping cost, condition, and link to the product page. During web scraping, your eBay scrapers need to target the container that wraps each listing, then extract data from child elements such as the title or price.
• Product listing pages: These are much more standardized and have remained relatively the same for several years. A single product page usually contains detailed information and elements such as the product title, price section, seller information, item condition, description, and images. It is important to note that much of this data is embedded in structured HTML elements or loaded dynamically with JavaScript. 
• Product variants: Most products include several variants such as size, and color. They also include conditions that are often shown as dropdowns or selectable buttons. Changing any of these variants may update the price and availability without reloading the page. When scraping eBay, your scraping services need to detect the variant options and loop through them to collect all possible combinations.
• Dynamic content: This eCommerce website has some parts that may load after the initial page loads. In this case, simple HTML-only web scraping may miss data unless JavaScript is executed. You need to use the right web scraping tools combo to be able to capture all the important dynamic content. 

Tools & Libraries for eBay Scraping

The three main tools that you can use for eBay scraping include Puppeteer, Playwright, and other Python tools. Let’s walk you through an overview of each of these tools: 

• Puppeteer: It is a Node.js library that controls a real browser. Using Puppeteer is ideal when collecting data from pages that rely heavily on JavaScript. When using this tool, you can actively interact with the pages, including clicking the product variants or scrolling. The main downside with Puppeteer is that it is slower than basic web scraping, but more reliable for complex target web pages.
• Playwright: It is a Python library that is similar to Puppeteer but more modern and flexible. It supports multiple browsers and better handling of dynamic content, making it a good choice for large or more advanced web scraping projects. Python Playwright is often the preferred choice when stability and scale are important.
• Python-based tools: Python has libraries such as Requests and BeautifulSoup, or Selenium for browser automation that can be used for data scraping. Using these Python tools is ideal for web scraping simple pages or when speed is important. These Python tools integrate with most data analysis tools and pipelines. The main downside with these Python tools is that they may struggle with heavily dynamic content unless combined with a browser-based tool.

Latest Working eBay Scraper Code (Quick Start)

Below is a ready-to-run Puppeteer example that fetches basic data about a listing on the search results page. Before running the code, here are a few requirements: 

• Node.js 18+ to effectively run Puppeteer.
• Puppeteer installed: You can install it using this command: npm install puppeteer.

Code snippet example below: 

JavaScript

importpuppeteer from"puppeteer";
construn = async() => {
  constbrowser = awaitpuppeteer.launch({ headless: true});
  constpage = awaitbrowser.newPage();
  constsearchUrl =
    "https://www.ebay.com/sch/i.html?_nkw=iphone+14";
  awaitpage.goto(searchUrl, { waitUntil: "networkidle2"});
  constlistings = awaitpage.evaluate(() => {
    returnArray.from(document.querySelectorAll(".s-item")).map(item => ({
      title: item.querySelector(".s-item__title")?.innerText || null,
      price: item.querySelector(".s-item__price")?.innerText || null,
      link: item.querySelector(".s-item__link")?.href || null,
    }));
  });
  console.log(listings.slice(0, 5));
  awaitbrowser.close();
};
run();

The above script does three simple things: It opens eBay, searches for a keyword, and extracts data, including titles, prices, and links. 

Create an eBay Web Scraper with Puppeteer

By using Puppeteer, you can easily scrape data from eBay and other Javascript-heavy websites. As discussed earlier, Puppeteer is a Node.js library that controls a real browser, making it ideal for web scraping JavaScript-heavy websites like eBay, where content loads dynamically and changes when users interact with the page.

Key Advantages of Scraping eBay with Puppeteer

• Speed: It renders JavaScript just like a real user’s browser.
• Robust: This tool can can click buttons, filters, and variants just like real humans.
• Dynamic content: Reliable for dynamic prices and listings.
• Ease of use: It is easy to debug by running in non-headless mode.

With this brief overview, let’s now dive into the six key steps for scraping eBay data using Puppeteer. For each step, we will provide a simple script that you can use to execute. 

Step 1: Launch Puppeteer and Open eBay

Use the script below:

JavaScript

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://www.ebay.com", { waitUntil: "networkidle2" });

The script above starts a browser session, opens a new tab, and loads the eBay homepage.

Step 2: Perform a Search Query

JavaScript

awaitpage.type("#gh-ac", "iphone 14");
awaitpage.click("#gh-btn");

The above script types a search term into the search box and submits the form, just like a real user.

Step 3: Wait for Results to Load

JavaScript

await page.waitForSelector(".s-item");

This part of the program pauses the script until search result listings are visible, ensuring the page is ready to scrape.

Step 4: Scrape Listing Data

JavaScript

const data = await page.evaluate(() => {
  return Array.from(document.querySelectorAll(".s-item")).map(item => ({
    title: item.querySelector(".s-item__title")?.innerText,
    price: item.querySelector(".s-item__price")?.innerText,
    shipping: item.querySelector(".s-item__shipping")?.innerText || "N/A",
  }));
});

Once the results are loaded, the script above reads the page’s HTML and extracts listing details such as title, price, and shipping info as configured. 

Step 5: Save Outputs to CSV

JavaScript

importfs from"fs";
constcsv =
  "title,price,shipping\n"+
  data
    .map(d => `"${d.title}","${d.price}","${d.shipping}"`)
    .join("\n");
fs.writeFileSync("ebay_listings.csv", csv);

This script converts the scraped data into CSV format and saves it to a file called ebay_listings.csv for later use.

Step 6: Close the Browser

JavaScript

await browser.close();

Finally, this script ends the browser session and frees system resources.

By implementing the six steps discussed above, you will have a working eBay scraper for collecting basic information from the website’s search results page. You can later add other capabilities such as pagination, proxies, or error handling to make your eBay scraper more robust. 

Build an eBay Scraper with Playwright (Python)

As covered in an earlier section, Playwright for Python is a modern browser automation tool that lets you control real browsers using popular engines like Chromium, Firefox, and WebKit. Just like Puppeteer, it also works very well for JavaScript-heavy sites such as eBay. However, it is often seen as more stable and flexible than Puppeteer, especially for larger web scraping projects.

Compared to Puppeteer, using Python Playwright has these key benefits: 

• Better waiting and auto-retry logic.
• Supports multiple browsers out of the box.
• Feels more reliable for dynamic pages.

Key Advantages of Scraping eBay with Python Playwright

• Handling dynamic content: It handles JavaScript-rendered content very well.
• Waiting capabilities: It has built-in waiting that reduces flaky scripts.
• Modern browser support: It works across Chromium, Firefox, and WebKit browsers.
• Ecosystem support: Has strong Python ecosystem support.

With this brief overview, let’s dive into the steps for web scraping with Python Playwright:

Step 1: Install Dependencies & Launch Python Playwright

Install the dependencies using this command: 

Bash

pip install playwright
playwright install

This command installs Playwright and its dependencies. 

Python

fromplaywright.sync_api importsync_playwright
playwright = sync_playwright().start()
browser = playwright.chromium.launch(headless=True)
page = browser.new_page()

The above Python code snippet starts a real browser session that your script can control.

Step 2: Define Selectors & Run a Search

search_url = "https://www.ebay.com/sch/i.html?_nkw=iphone+14"
page.goto(search_url)

The Python script above opens an eBay search results page and defines which HTML elements will be scraped. Selectors will later be used to target titles, prices, and links.

Step 3: Wait for Page Rendering

Python

page.wait_for_selector(".s-item")

This Python script ensures all listings are fully loaded before data extraction starts. The goal here is to prevent missing data caused by slow page loads.

Step 4: Extract Product Details & Variants

Python

items = page.query_selector_all(".s-item")
data = []
foritem initems:
    title = item.query_selector(".s-item__title")
    price = item.query_selector(".s-item__price")
    link = item.query_selector(".s-item__link")
    data.append({
        "title": title.inner_text() iftitle elseNone,
        "price": price.inner_text() ifprice elseNone,
        "link": link.get_attribute("href") iflink elseNone
    })

The Python script above loops through all visible listings and pulls key product information. This is also where you would extend logic to handle variants like condition or size if required.

Step 5: Save Extracted Data to CSV

Python

importcsv
withopen("ebay_listings.csv", "w", newline="", encoding="utf-8") asf:
    writer = csv.DictWriter(f, fieldnames=["title", "price", "link"])
    writer.writeheader()
    writer.writerows(data)

After the data has been extracted, the above Python code adds the ability to store this data in a CSV file so it can be reused later.

Step 6: Error Handling & Cleanup

Python Playwright can handle error using try as shown in the sample code below

Python

try:
    pass  # scraping logic runs here
finally:
    browser.close()
    playwright.stop()

The Python code above ensures the browser closes properly, even if something goes wrong to avoid hanging browser processes.

Step 7: Run the Script

After writing and saving your full script, it is time to run it. 

Bash

python ebay_scraper.py

Once the script is executed and complete, your CSV file will contain the extracted listings.

Scraping eBay Listings: What Data You Can Extract

Some of the common data that you can collect when scraping listings includes: 

• Product title: This is basally the main listing name.
• Price: The listing prices can include the Buy It Now price, bid price, or price range.
• Condition: The condition of the product can be new, used, refurbished, etc.
• Product variants: These can include size, color, storage, bundle options, etc.
• Availability: This metric represents items in stock or quantity left.
• Seller information: Details of the seller can include the seller name, rating, feedback count.
• Shipping details: Includes cost, delivery time, location.
• Images: These include the main image and gallery images.
• Sales signals: These are the number of items sold or watchers.

Scraping eBay Search Results at Scale

Scraping eBay or other websites at scale requires using an eBay scraper that can handle pagination since it has to move through multiple results pages to obtain all the relevant data. With this e-commerce website, searching results are often split across pages and each web page has page or _pgn parameter in the URL. 

So, your web scraper will run a specific search query, scrape all listings on page 1, and then move to page 2, page 3, and so on. It should then stop when no more listings are found.

How to Reveal & Scrape Hidden Web Data

Not all data on this site is visible in the raw HTML. Some data is loaded dynamically. Some of the common sources of such data include: 

• JavaScript-rendered content.
• Background API requests.
• Elements hidden in the DOM.
• Data embedded in JSON scripts.

To access this hidden, here are some techniques that you can use: 

• Use browser automation tools that run JavaScript.
• Wait for elements to appear before web scraping.
• Inspect network requests to find API endpoints.
• Parse embedded JSON inside <script> tags.

Scraping hidden web data is more effective when using browser-based tools than basic HTTP requests.

How to Avoid eBay Blocking Your Scraper

This e-commerce website actively detects automated web traffic and can block it. To prevent your GET requests from getting blocked, here are a few techniques that you should consider using: 

• Use proxies to rotate IP addresses after a given time interval or number of requests.
• Add delays between requests to make your traffic seem more human.
• Set realistic headers (User-Agent, language, timezone).
• Use real browsers instead of raw requests.

As you might have noticed, the goal is to make your traffic seem as human as possible. 

Scaling Up: Challenges & Limitations

As you scale your eBay scraping processes, you will encounter a number of challenges. Some of these include:

• Rate limits: Sending too many requests requests trigger blocks.
• CAPTCHAs: Sometimes pages ask for human verification.
• IP bans: Happens when the platforms detects repeated requests from the same IP.
• Concurrency issues: Using too many browsers slow systems.
• Costs: Web scraping involves using tools like proxies, servers, and compute resources, which are all costly.

Using BrowserQL for eBay Scraping at Scale

Using BrowserQL is one of the most effective ways to overcome the above challenges. It is a cloud-based browser automation solution designed for large-scale scraping. So, instead of running browsers locally, you run them remotely in a managed environment.

This approach is useful when scraping high volumes of pages, managing many sessions at once, and avoiding local resource limits. It also reduces the infrastructure complexity and improves stability.

Understanding eBay’s Bot Detection System

To bypass the platform’s bot detectors, you first of all need to know how they work. 

Advanced Bot Detection

Some of the techniques eBay uses to detect bots include: 

• Browser fingerprinting.
• Behavior analysis (mouse movement, timing).
• JavaScript challenges.
• Traffic pattern analysis.

Limitations of Stealth Plugins & Proxies

Stealth plugins alone are not enough to bypass eBay’s advanced bot detection system. Also, cheap or reused proxies can easily be detected and flagged.

How to Bypass eBay’s Bot Detection

• High-quality rotating proxies: Use advanced proxy services that include features like automatic IP rotation. 
• Browsers: Run real browsers with realistic behavior.
• Requests speeds: Limit the request speed and concurrency.
• Browser fingerprints: Rotate fingerprints and sessions.
• Use Managed browsers: These are the most effective when web scraping at scale. 

What Is BrowserQL?

It is a cloud-based web scraping solution that is designed for large-scale, browser-level data collection. Instead of running browsers on your local machine, BrowserQL runs real browsers for you in the cloud and returns the results. Using BrowserQL makes high-volumes easier and more reliable, especially on websites like eBay that use strong anti-bot systems.

How BrowserQL Works

BrowserQL runs real browsers in the cloud and executes all your web scraping logic remotely. It also handles browser fingerprints, sessions, and scaling. After scraping the data, it returns raw HTML or extracted data via an API. To get started, you need to send a GraphQL-style request, and then BrowserQL will handle the browser execution behind the scenes.

Setting Up BrowserQL for eBay Scraping

Follow these four simple steps to setup BrowserQL:

• Step 1: Create an account with Bright Data.
• Step 2: Get your BrowserQL API token.
• Step 3: Decide what data you want to extract. This can include listings, prices, sellers, etc.
• Step 4: Finally, write a BrowserQL query or mutation to load pages.

Building an eBay Scraper REST API Using BrowserQL (BQL)

BrowserQL allows you to create a REST-style API that can automatically scrape eBay whenever it is called. This scraper API is useful for dashboards, internal tools, or automation workflows. Let’s walk you through the steps for creating this API. 

Step 1: Setting Up Your Environment

The first step involves preparing your local project to send requests to BrowserQL. 

This includes: 

• Choosing a backend language (Python or Node.js)
• Storing your BrowserQL API token securely
• Installing an HTTP client such as Requests, axios, or fetch

Step 2: Configuring Your Script

Your script should define the target eBay URL and headers for the scrape. So, you will need to configure the search keyword or product URL, region or marketplace, and your preferred output format. The data format can be HTML or structured data such as CVS

Step 3: Writing the BrowserQL Mutation

Here is an example:

GraphQL

mutation {
  browserql(
    url: "https://www.ebay.com/sch/i.html?_nkw=iphone+14",
    render: true
  ) {
    html
  }
}

The above Python script tells BrowserQL what page to open and to return the rendered HTML.

Step 4: Sending the Request

The next step is sending the mutation to the BrowserQL API endpoint. Below is a simple script for sending the request: 

Python

importrequests
API_URL = "https://api.brightdata.com/browserql"
API_TOKEN = "YOUR_BROWSERQL_API_TOKEN"
mutation = """
mutation {
  browserql(
    url: "https://www.ebay.com/sch/i.html?_nkw=iphone+14",
    render: true
  ) {
    html
  }
}
"""
headers = {
    "Authorization": f"Bearer {API_TOKEN}",
    "Content-Type": "application/json"
}
response = requests.post(
    API_URL,
    headers=headers,
    json={"query": mutation}
)
data = response.json()
html = data["data"]["browserql"]["html"]

The above script sends a request to the BrowserQL API with your mutation and receives fully rendered HTML.

Step 5: Parsing the Returned HTML

BeautifulSoup Python library is used to extract the listing data from the HTML returned by BrowserQL.

Python

frombs4 importBeautifulSoup
soup = BeautifulSoup(html, "html.parser")
listings = []
foritem insoup.select(".s-item"):
    title = item.select_one(".s-item__title")
    price = item.select_one(".s-item__price")
    link = item.select_one(".s-item__link")
    seller = item.select_one(".s-item__seller-info-text")
    shipping = item.select_one(".s-item__shipping")
    listings.append({
        "title": title.get_text(strip=True) iftitle elseNone,
        "price": price.get_text(strip=True) ifprice elseNone,
        "link": link["href"] iflink elseNone,
        "seller": seller.get_text(strip=True) ifseller elseNone,
        "shipping": shipping.get_text(strip=True) ifshipping elseNone,
    })

The program above extracts listing data from the returned HTML.

Step 6: Save Data to CSV

Python

importcsv
withopen("ebay_listings.csv", "w", newline="", encoding="utf-8") asfile:
    writer = csv.DictWriter(
        file,
        fieldnames=["title", "price", "link", "seller", "shipping"]
    )
    writer.writeheader()
    writer.writerows(listings)

The script above saves extracted listing data into a CSV file named ebay_listings.csv.

FAQ: eBay Scraping Questions Answered

Is it legal to scrape eBay.com?

It is legal as long as you collect publicly available data. Remember to review the platform’s Terms of Service and avoid scraping private, user-only, or personal data. To prevent your web scrapers from getting blocked, don’t overload their servers or bypass security controls. 

How to crawl eBay efficiently?

To crawl efficiently, focus on speed, stability, and safety. Use pagination and reuse browser sessions to avoid sending too many requests. Also, consider adding delays between requests to make your traffic seem more natural. 

Does eBay offer an official API?

Yes, it does provide an official API that developers can use to access listings, orders, and seller data. Using this API requires developer registration and API keys. It also has limits and restrictions.