How to Feed Data to an LLM Using Python

LLMs do not have access to real-time information. They do not know what happened in the last hour. Sometimes you want to analyze data patterns in a file, which might take hours of manual job but if you feed that data to an LLM, you can generate a conclusion within minutes.

In this article, we will learn how you can feed live data scraped from a website using Scrapingdog to any LLM.

Why use Scrapingdog for feeding the data to an LLM?

Most LLM APIs can’t browse the web on their own. Even when they can, feeding them raw HTML is far from ideal. A typical webpage may contain 200KB of navigation menus, ads, JavaScript, and other boilerplate, while only 2KB contains the actual content you need. Passing that directly to an LLM wastes valuable context window space and makes it harder for the model to focus on relevant information.

On top of that, most LLMs simply make a standard HTTP request when fetching a webpage. They don’t handle JavaScript rendering, proxy rotation, CAPTCHAs, or other anti-bot protections, so many requests fail or return incomplete content. That’s where a web scraping API like Scrapingdog becomes essential.

Best Scraping API for Training LLMs

Prerequisite

Before we start writing our code, make sure we have these things in place.

You have to create an account on Scrapingdog. On signup, you will get 200 free credits.
Access to the Anthropic API.
I hope you already have Python 3.x on your machine. If not, then you can download it from here.
Install the requests library for making HTTP connections with Scrapingdog and Anthropic APIs.

Scraping the Data with Scrapingdog

For this article, we will feed live Google News data using the Google News API provided by Scrapingdog. You can learn more about the API by reading the documentation. You can even try the API directly from the dashboard.

You will get ready to use JSON data, which prevents extra token consumption. Click the Get Code button to copy the ready-to-use Python code.

1import requests
2 
3api_key = "your-api-key"
4url = "https://api.scrapingdog.com/google_news"
5 
6params = {
7    "api_key": api_key,
8    "query": "usa vs iran",
9    "country": "us",
10    "advance_search": "false",
11    "domain": "google.com"
12}
13 
14response = requests.get(url, params=params)
15 
16if response.status_code == 200:
17    data = response.json()
18    print(data)
19else:
20    print(f"Request failed with status code: {response.status_code}")

Now, we can feed this JSON data to any LLM model.

Feeding scraped data to an LLM

Before we feed this JSON to an LLM, we can trim it down even further. A query returns a news_results array where each item looks like this:

1{
2  "title": "US and Iran pause strikes for third night to make space for talks",
3  "snippet": "President Donald Trump has paused attacks on Iran for the third night in a row...",
4  "source": "BBC",
5  "lastUpdated": "2 days ago",
6  "url": "https://www.bbc.com/news/articles/c5y45kdkynpo",
7  "scrapingdog_link": "https://api.scrapingdog.com/scrape?api_key=...&url=...",
8  "imgSrc": "http://t0.gstatic.com/images?q=..."
9}

Fields like scrapingdog_link and imgSrc are useful for your app's UI but add nothing for the model; they just burn tokens. Strip the response down to what the LLM actually needs (title, snippet, source, lastUpdated, url) and flatten it into plain text:

1def format_for_llm(news_results):
2    entries = [
3        f"[{i + 1}] {item['title']}\n{item['snippet']}\n"
4        f"Source: {item['source']} ({item['lastUpdated']})\nURL: {item['url']}"
5        for i, item in enumerate(news_results)
6    ]
7    return "\n\n".join(entries)

This keeps token usage low and gives the model something readable instead of a nested JSON blob it has to mentally parse. Now, we can finally feed this data.

1def get_latest_news(query):
2    response = requests.get(
3        "https://api.scrapingdog.com/google_news/",
4        params={
5            "api_key": os.environ["SCRAPINGDOG_API_KEY"],
6            "query": query,
7        },
8    )
9    return response.json()["news_results"]
10 
11context = format_for_llm(get_latest_news("Iran US ceasefire talks"))
12 
13response = requests.post(
14    "https://api.anthropic.com/v1/messages",
15    headers={
16        "Content-Type": "application/json",
17        "x-api-key": os.environ["ANTHROPIC_API_KEY"],
18        "anthropic-version": "2023-06-01",
19    },
20    json={
21        "model": "claude-sonnet-4-6",
22        "max_tokens": 1000,
23        "messages": [
24            {
25                "role": "user",
26                "content": (
27                    "Using the following recent news articles, answer the question. "
28                    f"Cite the source for each fact.\n\n{context}\n\n"
29                    "Question: What's the current status of the Iran-US talks?"
30                ),
31            }
32        ],
33    },
34)

Let me explain this code in brief:

Fetches recent news from Scrapingdog’s Google News API based on a search query.

Formats the news articles into a clean text context suitable for an LLM.
Sends the context to Claude Sonnet 4.6 via Anthropic’s Messages API.
Prompts Claude to answer a specific question using only the provided news.
Requests source citations so each factual statement is backed by the original news articles.
Returns an AI-generated, up-to-date answer grounded in the latest news rather than the model’s training data.

This way, you can create a live data feed for any LLM.

You can use any Scrapingdog API as the data source, not just the Google News API. For example, you can use the Google Search API, Amazon Search API, or any of the other APIs too.

Conclusion

Feeding an LLM live data isn’t complicated once you have a clean source to pull from. Scrape it, trim it down to what the model actually needs, and pass it along with a clear question; that’s the whole loop. What used to take hours of manually reading through articles or spreadsheets now takes a few seconds and a well-formed prompt. And this pattern isn’t tied to news; the same three steps work with the Google Search API, Amazon Search API, or any other Scrapingdog endpoint, so you can build the same kind of live-data feed for whatever your LLM needs to know about.

How to Feed Data to an LLM Using Python (With Live Examples)