rafaelsideguide
fb758fa05e
go
2024-06-06 14:01:16 -03:00
Nicolas
0310da6729
Update rate-limiter.ts
2024-06-06 09:31:44 -07:00
Nicolas
01503c1fbf
Nick:
2024-06-06 09:29:25 -07:00
rafaelsideguide
b3cae4c858
adding js and testing twine
2024-06-06 13:27:31 -03:00
rafaelsideguide
bc1c1e5053
updating version to check if it runs
2024-06-06 11:41:01 -03:00
Rafael Miller
7686ad5702
Merge pull request #196 from mattjoyce/main
...
Python-SDK transitional build setup for pyproject.toml
2024-06-06 10:26:16 -03:00
Nicolas
525b4f2a83
Update rate-limiter.ts
2024-06-05 14:38:10 -07:00
Nicolas
d7f8208cdb
Update email_notification.ts
2024-06-05 13:53:31 -07:00
Nicolas
ec10eb09f3
Update credit_billing.ts
2024-06-05 13:22:03 -07:00
Nicolas
5991000d2b
Update credit_billing.ts
2024-06-05 13:21:15 -07:00
Nicolas
5683bb2cc8
Nick:
2024-06-05 13:20:26 -07:00
rafaelsideguide
164676c70a
bugfix screenshot for readme pages
2024-06-05 15:34:42 -03:00
rafaelsideguide
935406b96a
Merge branch 'main' into pr/196
2024-06-05 15:19:25 -03:00
Nicolas
b4c6819a54
Nick:
2024-06-05 11:11:09 -07:00
rafaelsideguide
0d51b11dcd
missing breaks
2024-06-05 15:02:28 -03:00
Rafael Miller
64423441b2
Merge branch 'main' into main
2024-06-05 14:44:29 -03:00
Nicolas
beb7526d1d
Update webhook.ts
2024-06-05 10:38:05 -07:00
Nicolas
1a16378fe8
Merge pull request #234 from JakobStadlhuber/feat/webhook-self-hosted
...
Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload
2024-06-05 10:25:05 -07:00
Nicolas
7cb14edec8
Nick:
2024-06-05 10:13:52 -07:00
Rafael Miller
9e000ded03
Merge branch 'main' into feat/better-gdrive-pdf-fetch
2024-06-05 14:07:56 -03:00
rafaelsideguide
ccc55127d6
Added scroll xpaths on fire-engine for handling readme docs
2024-06-05 11:48:41 -03:00
rafaelsideguide
b5045d1661
[feat] improved the scrape for gdrive pdfs
2024-06-04 17:47:28 -03:00
Nicolas
96257b7b17
Update handleCustomScraping.ts
2024-06-04 12:22:46 -07:00
Nicolas
674500affa
Nick:
2024-06-04 12:15:39 -07:00
rafaelsideguide
5ae4d1caf5
Update single_url.ts
2024-06-04 15:28:09 -03:00
Jakob Stadlhuber
9e5ddec207
Remove default webhook URL from .env.example
...
The default value for the SELF_HOSTED_WEBHOOK_URL in the .env.example file was removed to prevent unintentional exposure or usage. The users are now required to explicitly specify
2024-06-04 19:56:35 +02:00
Jakob Stadlhuber
6208f4207d
Add support for Self-Hosted Webhook URL Usage and added project_id into the webhook payload
...
This commit introduces the capability of using a Self-Hosted Webhook URL. The application now checks for a self-hosted URL before querying the database for the webhook settings. If a Self-Hosted Webhook URL is set in the environment variables, it will be used directly, diminishing unnecessary database queries.
2024-06-04 19:55:07 +02:00
rafaelsideguide
93f3098672
build files
2024-06-04 14:54:54 -03:00
rafaelsideguide
64a4338ff0
Update single_url.ts
2024-06-04 14:40:05 -03:00
Rafael Miller
02fe470e20
Merge pull request #148 from mendableai/nsc/improvemnts-fixes-misc
...
Better fallbacks for initial crawl start
2024-06-04 14:31:10 -03:00
Rafael Miller
665a40d9f4
Merge pull request #212 from mendableai/bugfix/partial-data-js-sdk
...
[Bug] Improved js response and test for getting partial_data
2024-06-04 14:05:23 -03:00
rafaelsideguide
1f4c6b7a87
Update package.json
2024-06-04 13:59:48 -03:00
Rafael Miller
19c67916d4
Merge pull request #211 from mendableai/fix/rename-variables
...
[Fix] Changed timeout parameter name on js sdk
2024-06-04 13:57:58 -03:00
Rafael Miller
f4f87b5374
Merge branch 'main' into bugfix/partial-data-js-sdk
2024-06-04 13:40:42 -03:00
rafaelsideguide
4e3a0495d7
updated version 0.0.12 -> 0.0.13
...
- [ ] publish
2024-06-04 12:03:55 -03:00
Rafael Miller
b80fb374e5
Merge branch 'main' into playwright-service-bug-222
2024-06-04 11:57:17 -03:00
rafaelsideguide
6920ec8a61
bugfixing. already on main
2024-06-04 11:05:50 -03:00
Nicolas
d91b725c6f
Update fly.toml
2024-06-04 00:41:15 -07:00
Nicolas
cbf8d79cce
Update pdfProcessor.ts
2024-06-04 00:13:37 -07:00
Nicolas
3fc9004ba8
Update fly.toml
2024-06-03 23:49:46 -07:00
Nicolas
2ea01f1456
Update single_url.ts
2024-06-03 23:42:39 -07:00
Nicolas
854d5b3cb3
Update single_url.ts
2024-06-03 23:32:55 -07:00
Nicolas
99059814a8
Nick:
2024-06-03 21:32:48 -07:00
Nicolas
918059ee9e
Merge branch 'main' into nsc/improvemnts-fixes-misc
2024-06-03 16:46:02 -07:00
Nicolas
38e583f66c
Update socialBlockList.test.ts
2024-06-03 16:44:23 -07:00
Nicolas
c69c89f838
Nick:
2024-06-03 16:42:42 -07:00
Nicolas
48d1ec05b2
Merge branch 'main' into nsc/improved-blocklist
2024-06-03 16:38:03 -07:00
Nicolas
d30ced4394
Merge pull request #221 from mendableai/nsc/fwd-header-auth
...
feat: Ability to forward headers to reliable providers for auth etc...
2024-06-03 16:33:40 -07:00
Romain Bruyère
4987f901d1
Merge branch 'mendableai:main' into main
2024-06-03 21:29:33 +02:00
rafaelsideguide
4100cc9223
Update index.test.ts
2024-06-03 16:29:16 -03:00
rombru
3ff91ddd1f
fix: use @ instead of # for default BULL_AUTH_KEY. hash mark is reserved for URI fragments.
2024-06-03 21:28:25 +02:00
rafaelsideguide
c1aed1360e
Update index.test.ts
2024-06-03 15:51:07 -03:00
rafaelsideguide
1fc3a15149
Update single_url.ts
2024-06-03 15:24:40 -03:00
Nicolas
fde522c3e1
Update single_url.ts
2024-06-02 20:23:45 -07:00
Matt Joyce
deefe65cbe
Change the way the playwright response is parsed
...
Was failing with a Type Error, but actually looked ok.
This fixes the type error, and stop scraper fallback.
2024-06-01 19:16:56 +10:00
Matt Joyce
14896a9fdd
Fix PLAYWRIGHT_MICROSERVICE_URL
...
It needs to end in html, otherwise scrape will 404
2024-06-01 19:03:16 +10:00
Matt Joyce
1eacad4ef3
Clarifying wait type and name
2024-06-01 18:53:03 +10:00
Matt Joyce
c516140bfb
Various Linting
...
Pylint
C0114: Missing module docstring
C0115: Missing class docstring
C0116: Missing function or method docstring
C0303: Trailing whitespace
Import ordering
2024-06-01 18:53:03 +10:00
Matt Joyce
2a39b5382b
Add timeout to class and provide default.
2024-06-01 18:52:42 +10:00
Nicolas
8cb62dde92
Update website_params.ts
2024-05-31 16:09:39 -07:00
Nicolas
3b8059edb6
Update single_url.ts
2024-05-31 15:43:06 -07:00
Nicolas
6bea803120
Nick:
2024-05-31 15:39:54 -07:00
Nicolas
2139129296
Nick: v12
2024-05-31 11:39:55 -07:00
Nicolas
260e31c68b
Merge branch 'nsc/new-pricing'
2024-05-30 16:08:31 -07:00
Nicolas
aa8133ca7f
Update load-testing-example.ts
2024-05-30 16:07:14 -07:00
Nicolas
0c115c6181
Merge pull request #216 from mendableai/nsc/new-pricing
...
feat: New pricing/limits changes
2024-05-30 15:36:59 -07:00
Nicolas
6860ace4af
Nick:
2024-05-30 15:07:49 -07:00
Nicolas
6ceb7ff50a
Nick:
2024-05-30 14:46:55 -07:00
Nicolas
33f10a7f91
Nick: fixes
2024-05-30 14:42:32 -07:00
Nicolas
ace46f340b
Nick: new limits, new pricing
2024-05-30 14:31:36 -07:00
Matt Joyce
5c4b3e8f8a
Initial pyproject.toml
...
This will enable building using 'python -m build', without impacting the utility of setup.py, also provide a base for other build tools and automation.
2024-05-30 21:48:40 +10:00
Matt Joyce
dec225d368
Move version to __init__.py
...
Setup.py does not need to be edited when building the package.
2024-05-30 21:48:40 +10:00
rafaelsideguide
2b763d848b
improved js response and test for getting partial_data
2024-05-30 08:44:38 -03:00
rafaelsideguide
5b8b6902e7
Update index.ts
2024-05-30 08:25:13 -03:00
Nicolas
6c939d534d
Nick: small refactor
2024-05-29 19:43:51 -07:00
Eric Ciarla
37915e11e8
Final push
2024-05-29 21:18:24 -04:00
Eric Ciarla
a0e404f94e
init commit
2024-05-29 18:56:57 -04:00
rafaelsideguide
ee9a2184e2
Added custom scraping conditions for readme docs
2024-05-29 13:39:43 -03:00
Nicolas
c20c38721d
Update index.test.ts
2024-05-28 17:17:20 -07:00
Nicolas
0f43a12906
Update index.test.ts
2024-05-28 17:17:12 -07:00
Nicolas
f53d25efac
Merge branch 'main' into nsc/wait-for-param
2024-05-28 12:56:28 -07:00
Nicolas
1b3547dcf2
Nick:
2024-05-28 12:56:24 -07:00
rafaelsideguide
71187b03a2
added timeout
2024-05-27 16:48:08 -03:00
rafaelsideguide
d5c83803cd
fixing idempotency test
2024-05-27 16:35:01 -03:00
rafaelsideguide
41c4ef6a82
dotenv was missing
2024-05-27 16:23:57 -03:00
rafaelsideguide
127d2db1dd
added js/ts sdk tests
2024-05-27 15:54:09 -03:00
rafaelsideguide
a9b68d95d8
Update test.py
2024-05-27 14:28:44 -03:00
rafaelsideguide
667d3e4c4f
Merge branch 'test-sdks' of https://github.com/mendableai/firecrawl into test-sdks
2024-05-27 14:23:39 -03:00
rafaelsideguide
19decd1062
fixing workflow
2024-05-27 14:21:33 -03:00
Rafael Miller
3c8edf683c
Merge branch 'main' into test-sdks
2024-05-27 14:15:18 -03:00
rafaelsideguide
63772ea711
added github action workflow
2024-05-27 14:14:00 -03:00
Nicolas
1ef307cb6f
Nick: checks
2024-05-27 10:01:12 -07:00
Nicolas
1de53cc4d0
Nick: fixes
2024-05-26 18:15:05 -07:00
Nicolas
efb821d63b
Merge branch 'main' into main
2024-05-26 18:12:23 -07:00
Nicolas
ed4226fd1f
Update setup.py
2024-05-26 18:11:54 -07:00
Nicolas
1bbfb98d7e
Merge pull request #186 from Keredu/main
...
Limit on /search is not deterministic
2024-05-26 18:08:16 -07:00
Nicolas
67a53a9ae0
Merge pull request #190 from simonha9/simonha9/improve-rate-limit-error-msg
...
Feat: Provide more details for 429 error msg
2024-05-26 18:07:42 -07:00
Nicolas
7e2df7bd5e
Update auth.ts
2024-05-26 18:07:21 -07:00
Nicolas
7948c6cee2
Nick: fixed pip issues
2024-05-26 18:03:37 -07:00
Matt Joyce
b061e12030
added python versions requirement
...
this is inline with requests module, a critical dependency
2024-05-26 11:37:47 +10:00
