From b32057ec890dc5b79ea7b05323820cf337afe68f Mon Sep 17 00:00:00 2001 From: chand1012 Date: Sun, 5 May 2024 12:03:42 -0400 Subject: [PATCH] Update SELF_HOST.md --- SELF_HOST.md | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/SELF_HOST.md b/SELF_HOST.md index 8d1d490..0deb543 100644 --- a/SELF_HOST.md +++ b/SELF_HOST.md @@ -1,6 +1,41 @@ # Self-hosting Firecrawl -Refer to [CONTRIBUTING.md](https://github.com/mendableai/firecrawl/blob/main/CONTRIBUTING.md) for instructions on how to run it locally. +First, clone this repository and copy `.env.example` to `.env`. +```bash +git clone https://github.com/mendableai/firecrawl.git +cd firecrawl +cp .env.example .env +``` -*This repository is currently in its early stages of development. We are in the process of merging custom modules into this mono repository. The primary objective is to enhance the accuracy of LLM responses by utilizing clean data. It is not ready for full self-host yet - we're working on it* +Then, edit the .env.example to have the correct values for your environment. +``` +## To turn on DB authentication, you need to set up supabase. +USE_DB_AUTHENTICATION=false +# ===== Optional ENVS ====== + +# Supabase Setup (used to support DB authentication, advanced logging, etc.) +SUPABASE_ANON_TOKEN= +SUPABASE_URL= +SUPABASE_SERVICE_TOKEN= + +# Other Optionals +TEST_API_KEY= # use if you've set up authentication and want to test with a real API key +SCRAPING_BEE_API_KEY= #Set if you'd like to use scraping Be to handle JS blocking +OPENAI_API_KEY= # add for LLM dependednt features (image alt generation, etc.) +BULL_AUTH_KEY= # +LOGTAIL_KEY= # Use if you're configuring basic logging with logtail +PLAYWRIGHT_MICROSERVICE_URL= # set if you'd like to run a playwright fallback +LLAMAPARSE_API_KEY= #Set if you have a llamaparse key you'd like to use to parse pdfs +SERPER_API_KEY= #Set if you have a serper key you'd like to use as a search api +SLACK_WEBHOOK_URL= # set if you'd like to send slack server health status messages +POSTHOG_API_KEY= # set if you'd like to send posthog events like job logs +POSTHOG_HOST= # set if you'd like to send posthog events like job logs +``` + +Once that's complete, you can simply run the following commands to get started: +```bash +docker compose up +``` + +This will run a local instance of Firecrawl which can be accessed at `http://localhost:3002`.