Google Custom Search API Python Tutorial

Many applications needs google custom search engine to perform web scraping. In this case you need to implement google custom search option in your application. So in Google Custom Search API Python Tutorial, you will learn to implement google custom search in your python application So let’s gets start without any delay.

As we all know that google search engine is most popular search engine. It is running in world’s largest web index. So if you search anything on google search then you get thousands of results in the order of their relevancy. So before moving to our main topic of this tutorial, we will need to understand some basic knowledge of google custom search and why any application need it. So let’s see it.

Contents

1 Google Custom Search API Python – Understanding The Basics Of Google Custom Search
2 Google Custom Search API Python – Getting Started
3 Python Program for Google Custom Search

Google Custom Search API Python – Understanding The Basics Of Google Custom Search

What Is Google Custom Search ?

Google Custom Search enables you to create a search engine for your website, your blog, or a collection of websites.
You can configure your engine to search both web pages and images.
Google custom search searches across a specified collection of sites or pages
The main feature of google custom search is that you can get a Tropical search engine that means you can restrict search result to a particular website or a set of some particular websites.

Use Cases Of Google Custom Search

There are two main use cases for Custom Search –

you can create a search engine that searches only the contents of one website (site search).
you can create a search engine that focuses on a particular topic from multiple sites.

Why Google Custom Search ?

Although there is many web scraping techniques available, so you might think why we need google custom search. So the answer is following –

So consider a scenario where you want to use search results in some of your projects. Then how can you fetch search results using your code. It is obvious that you will do web scraping, you will send a request to that particular url and then you will get html code and then you can scrap the useful informations from it by using any programming language. Now the issue is that sometime web scraping gets in trouble and not encouraged by google and you might even be thought as a bot and get blocked by the google search engine. And that’s why google custom search is applied in that scenarios.

Also Check : Wikipedia API Python – Scrapping Wikipedia With Python

Google Custom Search API Python – Getting Started

Now we will see how can we implement google custom search using python.

Custom Search Engine JSON API

The Custom Search JSON API lets you develop websites and applications to retrieve and display search results from Google Custom Search programmatically. With this API, you can use RESTful requests to get either web search or image search results in JSON format.

Custom Search JSON API can return results in JSON data format.

Create a project on Google Developers Console

The first thing that you have to do is to create a project on google developers console.

You will get something like this –

Now click on Select Project and below screen will be open.

Now if you want to use your existing project then click on Search projects and folders otherwise to create new project click on NEW PROJECT option.
Now enter your project name and click on create button.

Enable Custom Search API and generate API key

Now you have to enable custom search API. So to enable this, type custom search API on search bar, and then click on ENABLE button of the following page. And now your current project is able to interact with this particular API.

Now the next thing you have to do is to go to credentials and in credentials part you have to create some credentials using which you can call google custom search API. Just click on create credentials option.

Now you will get following screen.

Now in which API are you using option you have to select Custom Search API option.
And now click on API key and now you will get this screen.

In API Restrictions you have to select Custom Search API option and click on create button. Now you will get your API key.

Create A search Engine

Now next thing you have to do is to create a search engine. So go to this link and create a search engine.

Now click on add button and fill the required options.

You have to fill sites to search column. Here you can add the sites from where you want to get search result.
Now fill the column of name of the search engine. Here you have to fill the name of search engine which you want.
Now click on create button, your custom search engine will be created.

Here i have included two search engine www.simplifiedpython.net and www.quora.com.

Also Check : OpenWeatherMap API Python – Access Current Weather Data Of Any Location

So we have created API key and custom search engine successfully. Now we will see how to perform search using python code.

Python Program for Google Custom Search

In this section, we will see how can we get search results using python code so let’s do it.

Installing Module

The first thing you have to do is to install google-api-python-client module. To install this run the following command on your command prompt.


pip install google-api-python-client

pip install google-api-python-client

Python Code For Google Custom Search

Now write the following code on your python IDE(check best python IDEs).


from googleapiclient.discovery import build
import pprint

#define key
api_key = "Your API Key"
cse_key = "Your Custom search Engine Key"

resource = build("customsearch", 'v1', developerKey=api_key).cse()
result = resource.list(q='simplified python', cx=cse_key).execute()

pprint.pprint(result)

from googleapiclient.discovery import build

import pprint

#define key

api_key = "Your API Key"

cse_key = "Your Custom search Engine Key"

resource = build("customsearch", 'v1', developerKey=api_key).cse()

result = resource.list(q='simplified python', cx=cse_key).execute()

pprint.pprint(result)

What we Did?

First of all import build method from googleapiclient module and import pprint module.
Then enter API key and CSE key.
Now create a resource object through which you can send the request.
To create a resource object, you have to call build() function. And the first parameter will be service that is “customsearch”. The second parameter will be the version that is “V1”. And the last parameter will be developer key that is api_key. This will build a simple resource for you.
In order to create custom search resource object, you have to call cse() method.
Now you have to call the list() function that will return you search results.
list() function takes two parameter. The first one q that is query string and the second one is cx that is custom search engine ID. This will just create a request for you, it is not actually executing the request. So to execute the request, you have to call the execute() method.
Finally print the result. Here i am using pprint because search result’s data is JSON data and JSON data is not formatted so it’s hard to read and that’s why we need the pretty printed.
Now run the code and check the result. So the search result is as below.

You can see in output, results have only fetched from simplified python page not any other page.

Now if you want to get all the results that are fetched then write this code snippets.


for item in result['items']:
    print(item['title'], item['link'])

for item in result['items']:

print(item['title'], item['link'])

Now see the result, only title and link has been fetched.

You can also check the length of item in one API call by running this code.


print(len(result))

print(len(result))

How To Get Image Results Using Google Custom Search

Now we will learn how to get image results using google custom search. To do this you have to add only one thing in above code.


from googleapiclient.discovery import build
import pprint
api_key = "Your API key"
cs_key = "Your cse key"

resource = build("customsearch", 'v1', developerKey=api_key).cse()

result = resource.list(q='python', cx=cs_key, searchType='image').execute()


for item in result['items']:
    print(item['title'], item['link'])

from googleapiclient.discovery import build

import pprint

api_key = "Your API key"

cs_key = "Your cse key"