0
Commit Graph

348 Commits

Author SHA1 Message Date
Nicolas
6fc1ee32fd
Merge pull request #275 from mendableai/feat/issue-273
Added pageOptions.removeTags
2024-06-13 13:27:01 -07:00
rafaelsideguide
676d6e8ab5 Added pageOptions.removeTags 2024-06-13 10:51:05 -03:00
Nicolas
182f8d4d6c Update index.ts 2024-06-12 18:07:05 -07:00
Nicolas
67dc46b454 Nick: clusters 2024-06-12 17:53:04 -07:00
rafaelsideguide
d20af257ba Added jobId to webhook data 2024-06-12 15:38:41 -03:00
rafaelsideguide
01c9f071fa fixed 2024-06-12 11:27:06 -03:00
rafaelsideguide
dc6acbf1f0 Merge remote-tracking branch 'origin/main' into feat/allowbackwardcrawling-option 2024-06-12 11:01:05 -03:00
Nicolas
f93231499f
Merge pull request #265 from mendableai/feat/issue-264
[Feat] Added route to clean completed jobs and a github action cron that triggers every 24h
2024-06-11 21:33:52 -07:00
Nicolas
45dee63943
Merge pull request #262 from mendableai/nsc/webhook-self-host-fix
Only fetch webhook from db if self host webhook not set and using db auth
2024-06-11 15:46:57 -07:00
rafaelsideguide
157fbe4a1e added bull auth key 2024-06-11 17:52:01 -03:00
rafaelsideguide
df3a678cf4 getting back the cancel test, this should work 2024-06-11 17:46:56 -03:00
rafaelsideguide
def2ba9987 added tests 2024-06-11 17:46:25 -03:00
Nicolas
1e3e06a1d5 Update replacePaths.test.ts 2024-06-11 13:02:39 -07:00
Nicolas
2239e03269 Update replacePaths.test.ts 2024-06-11 12:54:02 -07:00
Nicolas
520739c9f4 Nick: fixed bugs associated with absolute path replacements 2024-06-11 12:43:16 -07:00
rafaelsideguide
ee282c3d55 Added allowBackwardCrawling option 2024-06-11 15:24:39 -03:00
rafaelsideguide
a9f93c2f1e Added route to clean completed jobs and a github action cron that triggers every 24h 2024-06-11 14:18:05 -03:00
Nicolas
f6b06ac27a Nick: ignoreSitemap, better crawling algo 2024-06-10 18:12:41 -07:00
Nicolas
1bd0327e1a Merge branch 'main' into nsc/pageoptions-crawler 2024-06-10 17:15:10 -07:00
Nicolas
99f2ffd6d5 Update webhook.ts 2024-06-10 17:03:10 -07:00
Nicolas
7ae9778642 Update single_url.ts 2024-06-10 16:57:31 -07:00
Nicolas
913c1dd568 Nick: fetch -> axios and fix timeouts 2024-06-10 16:49:03 -07:00
Nicolas
3091f0134c Nick: 2024-06-10 16:27:10 -07:00
Nicolas
f24ca76618 Nick: removing rate limit emails for now 2024-06-07 10:39:11 -07:00
Nicolas
98d82c4cec Update search.ts 2024-06-06 20:02:21 -07:00
Nicolas
5e80f8af87 Nick: llm extract 50 2024-06-06 18:35:44 -07:00
rafaelsideguide
f5318ea7d7 Update index.test.ts 2024-06-06 16:50:20 -03:00
rafaelsideguide
cd7f9abcec Update index.test.ts 2024-06-06 16:44:46 -03:00
rafaelsideguide
7b9b668b95 Update index.test.ts 2024-06-06 16:36:51 -03:00
rafaelsideguide
82e0ed4cd3 Update index.test.ts 2024-06-06 16:33:27 -03:00
rafaelsideguide
dac7612be2 Merge branch 'main' of https://github.com/mendableai/firecrawl into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 16:07:25 -03:00
Nicolas
c2ad358390 Nick: 2024-06-06 12:05:20 -07:00
rafaelsideguide
79ec9f04dc Merge branch 'main' of https://github.com/mendableai/firecrawl into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:58:14 -03:00
Nicolas
de06b13deb Update rate-limiter.ts 2024-06-06 11:56:22 -07:00
Nicolas
27a8fd0c3c Update rate-limiter.ts 2024-06-06 11:56:00 -07:00
Nicolas
1129d33321 Update rate-limiter.ts 2024-06-06 11:53:12 -07:00
rafaelsideguide
b234b4be5a Merge branch 'main' into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:44:29 -03:00
rafaelsideguide
af0bfca847 Merge branch 'main' into 194-sdk-ci-pipeline-for-publishing-pythonnode-sdk 2024-06-06 15:36:28 -03:00
rafaelsideguide
8132f22c73 nice 2024-06-06 15:36:20 -03:00
Nicolas
f1b5ec8517 Nick: fixes 2024-06-06 11:23:10 -07:00
Nicolas
deae7dcd61 Update email_notification.ts 2024-06-06 10:41:54 -07:00
Nicolas
f725fa5a97 Update email_notification.ts 2024-06-06 10:41:23 -07:00
Nicolas
0310da6729 Update rate-limiter.ts 2024-06-06 09:31:44 -07:00
Nicolas
01503c1fbf Nick: 2024-06-06 09:29:25 -07:00
Nicolas
525b4f2a83 Update rate-limiter.ts 2024-06-05 14:38:10 -07:00
Nicolas
d7f8208cdb Update email_notification.ts 2024-06-05 13:53:31 -07:00
Nicolas
ec10eb09f3 Update credit_billing.ts 2024-06-05 13:22:03 -07:00
Nicolas
5991000d2b Update credit_billing.ts 2024-06-05 13:21:15 -07:00
Nicolas
5683bb2cc8 Nick: 2024-06-05 13:20:26 -07:00
rafaelsideguide
164676c70a bugfix screenshot for readme pages 2024-06-05 15:34:42 -03:00