Search like a pro with the Google Custom Search Engine

#1

Hello fellow dashdashers :wave:

Let me introduce you to the holy grail of scraping a.k.a the Google Custom Search Engine (GCSE). It is a super powerful tool that can help you generate sales leads or support your talent acquisition process :star_struck:

These are the steps:

  1. Get your Google Cloud Platform API key
  2. Setup a Google Custom Search Engine
  3. Query your searches via the GCSE integration
  4. Build cool stuff! :construction_worker_man::construction_worker_woman:

:one: Get your GCP API key
This one is easy, thanks to @guven2.0 :wink: Follow the documentation here.

:two: Setup a Google Custom Search Engine
Navigate to the GCSE login page, sign in and be ready to create your very first custom search engine.

Entering a site to search is mandatory in this step. You can delete your choice later, ultimately enabling you to search the entire web. So go ahead and enter any website or multiple ones that you’d like to search on :mag::wink:

In your search engine’s setup page you can fine tune the settings and find the ID for this custom search engine. Store it safely—you’ll need it later.

If you’d like to search the entire web, don’t forget to delete any sites you might have previously entered and toggle the search the entire web button.

:three: Query via the GCSE integration

Navigate to the integrations dashboard and connect to the GCSE integration. Use the specific GCSE function to query Google and get back search results as a JSON object :man_technologist::woman_technologist:

A1 = 01663930225133...
A2 = automobile manufacturers Germany
A4 = CUSTOM_SEARCH_GOOGLE(A2,A1)

In the JSON object, you can find your search results via [items:]. In this case, the fourth search result ended up being the Volkswagen Group :oncoming_automobile:

:four: Build cool stuff! :construction_worker_man::construction_worker_woman:

The possibilities are endless! Here are some use cases we have already documented for you:

:vulcan_salute:


:five: Alternatively: Query your searches via GET requests

If you don’t want to use the GCSE integration, you will have to build a GET request in order to pull data from the API. Within the GCSE API documentation we can find the standard URL with the mandatory parameters:

https://www.googleapis.com/customsearch/v1?KEY=[YOUR API KEY]&CX=[CSE ID]&Q=[YOUR QUERY]

There a further optional parameters, which you can find here.

Off to the fun part, let’s put everything together and send out the GET request :muscle::nerd_face:

A1 = AIzaSyDSh7q...
A2 = 01663930225133...
A3 = automobile manufacturers Germany
A4 = GET("https://www.googleapis.com/customsearch/v1?KEY="&A1&"&CX="&A2&"&Q="&A3&"")
3 Likes
Increasing Count to be > 10
*Update*: AVERAGE, MEDIAN and more Integration functions
Generate leads through LinkedIn