| # `SERPent` |
|
|
| ## SERP results scrapping |
|
|
| SERPent exposes an unified API to query SERP (Search Engine Result Pages) for a few common search engines, namely: |
|
|
| - DuckDuckGo |
| - Brave |
| - Bing |
| - Google Patents |
| - arXiv |
| - Google |
|
|
| The application uses the `playwright` library to control a headless web browser, to simulate normal user activity, to fool the anti-bot measures often present on those sites. See the `/serp/` endpoints for search results scrapping. |
|
|
|
|
| ## Website sources scrapping |
|
|
| SERPent also exposes a few endpoints to scrap the contents of certain sources (patents, scholar). See the `/scrap/` endpoints for supported website sources scrapping. |
|
|
|
|
|
|
|
|
