Python Web Scraping: Sending User-Agent Headers

To send a user-agent while scraping using Python, you can use the 'requests' library and pass the user-agent as a header in the request. Here's an example:

import requests

user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'

url = 'https://example.com'

headers = {'User-Agent': user_agent}

response = requests.get(url, headers=headers)

print(response.content)

In this example, we define a 'user_agent' string that represents the browser and operating system of the client. We then create a 'headers' dictionary and pass the 'user_agent' string as the value for the 'User-Agent' key. Finally, we make the request to the desired URL using the 'requests.get()' method and passing the 'headers' dictionary as an argument. The response content is printed to the console.

By sending a custom User-Agent header, you can mimic a real browser and potentially avoid being blocked by websites that have anti-scraping measures in place.