Python Web Scraping: How to Send a User-Agent Header

To send a User-Agent header while scraping in Python, you can use the 'requests' library. The User-Agent header tells the website which browser or program is making the request. Some websites may block requests that do not have a User-Agent header or that have a suspicious User-Agent. Here's an example code snippet:

import requests

# Set the User-Agent header to a fake Chrome browser
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}

# Make a GET request to a website
response = requests.get('https://www.example.com', headers=headers)

# Print the HTML content of the website
print(response.content)

In this example, we set the User-Agent header to a fake Chrome browser. You can change the User-Agent string to any other browser or program that you want to mimic. The 'headers' parameter is passed to the 'get' method of the 'requests' library. The 'response' object contains the HTML content of the website, which we print to the console.