Guides

Source-data guides

Practical guides for crawler decisions, website-to-RAG preparation, document processing, and source-data workflows.

Start here
Run the crawlerCollect public web content with scoped search, scrape, crawl, estimates, and export-ready output.Open Plan a RAG pipelineUse crawler and document output as reviewable source material before chunking, embeddings, and retrieval.Open Prepare documentsThink through extraction, normalization, metadata, and evidence before processed files become AI input.Open Review security postureKeep crawler and source-data workflows bounded, auditable, and controlled before scaling paid usage.Open
Guide library
RAG preparation guide

How to prepare website content for RAG

A practical guide to preparing website content for RAG workflows with scoped crawling, clean exports, source review, chunking, and retrieval-ready structure.

  • Use explicit URL targets and same-site limits before crawling.
  • Export source material in reviewable formats before embedding.
  • Keep source evidence so bad answers can be traced back to bad inputs.
Read guide
Crawler vs ETL guide

Web crawler vs. ETL pipeline: what is the difference?

Understand how web crawlers and ETL pipelines differ, when to use each one, and why SourceOfTruth.io keeps crawler collection separate from broader AI data preparation.

  • Use a crawler when the source is a website or documentation set.
  • Use ETL when data needs repeatable extraction, transformation, and loading across systems.
  • Crawler jobs need limits, estimates, and export evidence.
Read guide
Document processing guide

AI document processing: from files to usable source text

A source-data guide to AI document processing, including extraction, metadata, cleaning, chunking, evidence review, and metered downstream processing.

  • Extract text before applying AI analysis.
  • Keep metadata and source evidence attached to processed output.
  • Review extraction quality before chunking and embedding.
Read guide
RAG data pipeline guide

What belongs in a RAG data pipeline?

A practical overview of RAG data pipeline stages: source collection, cleaning, chunking, embeddings, indexing, retrieval testing, and evidence review.

  • The source layer determines the quality ceiling for RAG.
  • Chunking and embeddings should happen after source cleanup.
  • Retrieval tests need source evidence, not just answer demos.
Read guide
Crawler comparison guide

Looking for a Firecrawl alternative for source-data workflows?

A practical guide for teams evaluating crawler tools for AI source-data workflows, clean exports, metered usage, and RAG preparation.

  • Evaluate crawler tools by workflow fit, not just raw scrape ability.
  • Look for clean exports, limits, estimates, and job history.
  • Keep crawler pricing separate from advanced AI processing costs.
Read guide
Definitions

Crawler-first launch

Search + Web Crawler and supported document workflows are active customer surfaces; advanced/provider-dependent capabilities remain separately gated.

RAG preparation

Source material should be reviewed, cleaned, and structured before it becomes chunks, embeddings, and retrieval context.

Document processing

Supported document extraction, normalization, Transform, and indexing workflows are available; broader automation and advanced processing remain separately gated.

ETL/ELT roadmap

Supported document ETL/Transform is active. Broader connector-driven ETL/ELT automation remains roadmap work pending connector, retry, governance, and launch acceptance.

Crawler and supported document workflows are active.Guides should distinguish current supported document workflows from broader connector-driven or advanced pipeline capabilities that remain gated.