Nicolas
|
f6b06ac27a
|
Nick: ignoreSitemap, better crawling algo
|
2024-06-10 18:12:41 -07:00 |
|
Nicolas
|
1bd0327e1a
|
Merge branch 'main' into nsc/pageoptions-crawler
|
2024-06-10 17:15:10 -07:00 |
|
Nicolas
|
7ae9778642
|
Update single_url.ts
|
2024-06-10 16:57:31 -07:00 |
|
Nicolas
|
913c1dd568
|
Nick: fetch -> axios and fix timeouts
|
2024-06-10 16:49:03 -07:00 |
|
Nicolas
|
3091f0134c
|
Nick:
|
2024-06-10 16:27:10 -07:00 |
|
rafaelsideguide
|
164676c70a
|
bugfix screenshot for readme pages
|
2024-06-05 15:34:42 -03:00 |
|
Nicolas
|
b4c6819a54
|
Nick:
|
2024-06-05 11:11:09 -07:00 |
|
rafaelsideguide
|
0d51b11dcd
|
missing breaks
|
2024-06-05 15:02:28 -03:00 |
|
Nicolas
|
7cb14edec8
|
Nick:
|
2024-06-05 10:13:52 -07:00 |
|
Rafael Miller
|
9e000ded03
|
Merge branch 'main' into feat/better-gdrive-pdf-fetch
|
2024-06-05 14:07:56 -03:00 |
|
rafaelsideguide
|
ccc55127d6
|
Added scroll xpaths on fire-engine for handling readme docs
|
2024-06-05 11:48:41 -03:00 |
|
rafaelsideguide
|
b5045d1661
|
[feat] improved the scrape for gdrive pdfs
|
2024-06-04 17:47:28 -03:00 |
|
Nicolas
|
96257b7b17
|
Update handleCustomScraping.ts
|
2024-06-04 12:22:46 -07:00 |
|
Nicolas
|
674500affa
|
Nick:
|
2024-06-04 12:15:39 -07:00 |
|
rafaelsideguide
|
5ae4d1caf5
|
Update single_url.ts
|
2024-06-04 15:28:09 -03:00 |
|
rafaelsideguide
|
64a4338ff0
|
Update single_url.ts
|
2024-06-04 14:40:05 -03:00 |
|
Rafael Miller
|
02fe470e20
|
Merge pull request #148 from mendableai/nsc/improvemnts-fixes-misc
Better fallbacks for initial crawl start
|
2024-06-04 14:31:10 -03:00 |
|
Rafael Miller
|
b80fb374e5
|
Merge branch 'main' into playwright-service-bug-222
|
2024-06-04 11:57:17 -03:00 |
|
rafaelsideguide
|
6920ec8a61
|
bugfixing. already on main
|
2024-06-04 11:05:50 -03:00 |
|
Nicolas
|
cbf8d79cce
|
Update pdfProcessor.ts
|
2024-06-04 00:13:37 -07:00 |
|
Nicolas
|
2ea01f1456
|
Update single_url.ts
|
2024-06-03 23:42:39 -07:00 |
|
Nicolas
|
854d5b3cb3
|
Update single_url.ts
|
2024-06-03 23:32:55 -07:00 |
|
Nicolas
|
99059814a8
|
Nick:
|
2024-06-03 21:32:48 -07:00 |
|
Nicolas
|
918059ee9e
|
Merge branch 'main' into nsc/improvemnts-fixes-misc
|
2024-06-03 16:46:02 -07:00 |
|
Nicolas
|
38e583f66c
|
Update socialBlockList.test.ts
|
2024-06-03 16:44:23 -07:00 |
|
Nicolas
|
c69c89f838
|
Nick:
|
2024-06-03 16:42:42 -07:00 |
|
Nicolas
|
48d1ec05b2
|
Merge branch 'main' into nsc/improved-blocklist
|
2024-06-03 16:38:03 -07:00 |
|
Nicolas
|
d30ced4394
|
Merge pull request #221 from mendableai/nsc/fwd-header-auth
feat: Ability to forward headers to reliable providers for auth etc...
|
2024-06-03 16:33:40 -07:00 |
|
rafaelsideguide
|
1fc3a15149
|
Update single_url.ts
|
2024-06-03 15:24:40 -03:00 |
|
Nicolas
|
fde522c3e1
|
Update single_url.ts
|
2024-06-02 20:23:45 -07:00 |
|
Matt Joyce
|
deefe65cbe
|
Change the way the playwright response is parsed
Was failing with a Type Error, but actually looked ok.
This fixes the type error, and stop scraper fallback.
|
2024-06-01 19:16:56 +10:00 |
|
Nicolas
|
8cb62dde92
|
Update website_params.ts
|
2024-05-31 16:09:39 -07:00 |
|
Nicolas
|
3b8059edb6
|
Update single_url.ts
|
2024-05-31 15:43:06 -07:00 |
|
Nicolas
|
6bea803120
|
Nick:
|
2024-05-31 15:39:54 -07:00 |
|
Nicolas
|
6c939d534d
|
Nick: small refactor
|
2024-05-29 19:43:51 -07:00 |
|
Eric Ciarla
|
37915e11e8
|
Final push
|
2024-05-29 21:18:24 -04:00 |
|
Eric Ciarla
|
a0e404f94e
|
init commit
|
2024-05-29 18:56:57 -04:00 |
|
rafaelsideguide
|
ee9a2184e2
|
Added custom scraping conditions for readme docs
|
2024-05-29 13:39:43 -03:00 |
|
Nicolas
|
1b3547dcf2
|
Nick:
|
2024-05-28 12:56:24 -07:00 |
|
Nicolas
|
e98434606d
|
Update blocklist.ts
|
2024-05-24 15:04:15 -07:00 |
|
Nicolas
|
e5c8719554
|
Update blocklist.ts
|
2024-05-24 14:53:04 -07:00 |
|
Nicolas
|
53a214cefb
|
Merge pull request #168 from mendableai/nsc/allowed-keywords-in-blocklist
feat: Allow privacy/legal/ other pages in social media websites
|
2024-05-24 09:43:15 -07:00 |
|
rafaelsideguide
|
f4a3469b9e
|
Merge branch 'main' into bug/crawl-limit
|
2024-05-22 14:27:28 -03:00 |
|
Nicolas
|
0d187f0425
|
Merge pull request #77 from tractorjuice/patch-1
Add additional file extensions to crawler.ts
|
2024-05-22 10:16:49 -07:00 |
|
Nicolas
|
a8ff295977
|
Update single_url.ts
|
2024-05-21 18:50:42 -07:00 |
|
Nicolas
|
a5e718b084
|
Nick: improvements
|
2024-05-21 18:34:23 -07:00 |
|
Nicolas
|
6285f12cd1
|
Merge pull request #167 from mendableai/nsc/hyper-dx-integration
feat: HyperDX Integration
|
2024-05-21 13:19:38 -07:00 |
|
Nicolas
|
7f64fe884a
|
Update blocklist.ts
|
2024-05-20 17:26:01 -07:00 |
|
Nicolas
|
756f54466d
|
Nick: allowed keywords for now
|
2024-05-20 17:24:21 -07:00 |
|
Nicolas
|
77a79b5a79
|
Nick: max num tokens for llm extract (for now) + slice the max
|
2024-05-20 17:07:38 -07:00 |
|
