What is a web crawler, really? | Search Off the Record

Опубликовано 14 марта 2024, 13:13

In this episode of Search Off The Record, Gary Illyes and Lizzi Sassman take a deep dive into crawling the web: what is a web crawler, and how does it really work? Listen along as the Search team is joined by an expert web developer in the SEO community, Dave Smart, for an in-depth and technical discussion of all things crawling, and maybe dispel some myths along the way.



Resources:
Episode transcript → goo.gle/sotr070-transcript

Managing your crawl budget → goo.gle/3IzRZxl
Dave Smart on LinkedIn → goo.gle/3wPSuRA
Tame the Bots → goo.gle/4cfCQ1P
Search Central Help Forum → goo.gle/sc-forum
Indexing API docs → goo.gle/3v8yVU0

Watch more Search Off the Record → goo.gle/sotr-yt
Subscribe to Google Search Central Channel → goo.gle/SearchCentral

Search Off The Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.


#SOTRpodcast


Speaker: Gary Illyes, Lizzi Sassman