Gemini Importing Documents

The Gemini extension imports the same managed document catalogue into two independent destinations: Gemini File Search for semantic RAG, and the App database for model-free Website Search. One confirmed import updates both. See Overview & Setup for enabling the extension and creating a File Store.

Text, Markdown, HTML, and Razor files can feed the local Search index. HTML is converted to Markdown before indexing. Binary Office and PDF files can be sent to Gemini but need text conversion to participate in local Search.

Upload files and ZIP archives​

The upload drop zone accepts PDF, Markdown/MDX, plain text, HTML, reStructuredText, AsciiDoc, CSV, JSON, YAML, and ZIP archives containing supported files.

HTML files are converted to Markdown before they are cached, indexed, and uploaded. Razor .cshtml files use the same conversion after removing server-only lines whose first non-whitespace character is @, and code blocks beginning with { through the matching } at the same or lower indentation. This keeps rendered page content searchable without leaking Razor directives into result snippets. When an HTML document has no explicit title, its generated display name uses the .md extension.

Choose an optional Destination category before uploading. A ZIP is expanded safely and each supported entry becomes a document; its internal directories are preserved beneath that category. For example, guides/auth/login.md uploaded to products becomes:

products/guides/auth/login.md

Hidden system files, macOS metadata, dependencies, build output, and other common archive noise are excluded automatically.

Import a local folder​

Folder import scans a directory on the machine hosting AI Chat. It is designed for sources you expect to preview and synchronize repeatedly.

Categories come from a document's directory, not its filename. Given docs/guides/auth/login.md, a Category root of docs produces guides/auth. Adding a destination of products produces products/guides/auth.

Preview before committing​

Preview import saves the current source settings and scans without changing documents, uploading content, or spending on embeddings. A dismissed preview leaves the folder in Saved imports.

Only Import N documents applies the preview. Progress updates naturally-for example, Uploading 16/21 documents to docs.example.com…-and View uploads opens Explorer at the destination category sorted by active uploads. Pending uploads resume when the application starts again after an interruption. The same confirmed import queues each changed document for the local Search index, so one synchronization updates both Website Search and Gemini RAG.

Saved imports​

Folder imports are saved when previewed; web crawls are saved after crawling. Both appear in the same Saved imports list. Give each source a unique name within its File Store. A dismissed preview leaves its source available to run later.

Loaded folder manifests open in Folder; crawl manifests open in Web crawl. Import folder continues from a crawl into Folder to configure attributes before Run import. Edit crawl settings and Edit folder settings move between those forms without losing their shared configuration. Preview and Run save current settings first. Selecting another store category does not overwrite the loaded import's destination.

Removing and reloading the same manifest in the same store reconnects its existing documents. Loading a crawl workspace outside your private Gemini imports directory copies its manifest, generated Markdown, and nested metadata into a private named workspace; the original stays untouched. Reloading it refreshes that same private copy without creating suffixed duplicates.

Re-running compares normalized content, metadata, and titles, reusing document identities. Unchanged, successfully uploaded files need no new embeddings. Changed documents replace their old remote copy after the new copy succeeds; removal failures retain the old identity and report an error for retry. Pending and failed uploads reuse existing document rows. A locally unchanged file can still be Awaiting Gemini upload: use Resume or Run to finish its existing queue entry.

Missing upstream files follow the saved deletion policy; the default removes the Gemini copy and retains a local tombstone. Large deletions still require confirmation through the individual source run.

Saved imports are composable: run several of them into the same File Store to create a unified corpus from content that remains owned and deployed independently. For example, the Explorer below contains five saved imports spanning docs.servicestack.net, servicestack.net, react-templates.net and sharpscript.net. Each source keeps its own synchronization definition, metadata rules and canonical Source URLs, while customers get one Search and Assistant experience across all four sites.

Re-running a saved import compares normalized content and metadata independently. The saved source key identifies the same document on later runs, while content and metadata hashes determine whether it changed. Unchanged, successfully indexed files need no new work; locally unchanged files may still await a Gemini upload. Content changes and metadata-only changes queue both indexes because Gemini cannot patch indexed metadata in place and Search ranking or filtering may depend on the changed metadata.

When an upstream file disappears, the import removes its Gemini copy and retains a local removed upstream tombstone so the change remains visible. A deletion safety rail refuses an unexpectedly large removal, protecting against a mistyped path or incomplete source listing.

Trusted import folders​

Server-side folder access is privileged:

• Administrators may import any accessible folder.
• Other users are restricted to server-allowed directories and Gemini Trusted import folders.
• Real paths are checked so a symlink cannot escape an allowed root.

Trusted roots are saved in App_Data/chat/user/default/config.json. They can use any directory aliases registered in ChatFeature.AliasedDirectories, and can also be configured in that file:

{
  "gemini": {
    "importRoots": ["$WORKSPACE/docs", "/srv/knowledge"]
  }
}

For example, an App can define the $WORKSPACE alias used above when registering AI Chat:

services.AddPlugin(new ChatFeature {
    AliasedDirectories = {
        ["$WORKSPACE"] = "/srv/workspace",
    },
});

Change detection and recovery​

Saved imports retain SourceId, SourceKey, source ETag/updated time, content hash, metadata hash, and extractor version per document. A later run classifies each source as unchanged, added, updated, metadata-only, missing, or failed. Content changes replace both the Gemini document and its local Search sections; metadata-only changes avoid unnecessary content work.

Both the upload and Search workers use durable desired/completed state. Pending work survives an App restart and resumes automatically. Failed documents retain their error for inspection and retry. Use Sync Store to reload enabled saved imports, refresh crawls, import changed sources, and reconcile remote state, and Rebuild index when extraction or database search configuration changes - see Explore & Ask and Operations & Troubleshooting.

Versioned import configuration​

A folder or ZIP may include import.json. A root manifest supplies global defaults; a manifest in a nested directory inherits and overwrites settings for the files beneath it.

{
  "version": 1,
  "metadata": {
    "defaults": {
      "product": "ServiceStack",
      "status": "published",
      "tags": ["docs"]
    },
    "rules": [
      {
        "match": "auth/**/*.md",
        "set": { "tags": ["auth"] }
      }
    ]
  },
  "transforms": [
    {
      "match": "**/*.md",
      "pattern": "\\nEdit this page.*$",
      "replacement": "",
      "flags": "gim"
    }
  ]
}

When the UI contains no explicit metadata, Preview import automatically loads the root manifest. Saving a folder import writes its effective settings back atomically while preserving crawl and transform settings. Saving does not run the import.

Metadata defaults and rules in a manifest use the fields described in Metadata & Source URLs. To import a public website, stage it first with Crawling Websites, which hands its cleaned workspace to Folder import.

Portable source settings​

A root import.json can store the complete source alongside metadata:

{
  "version": 1,
  "source": {
    "name": "Product docs",
    "type": "folder",
    "config": {
      "path": ".",
      "include": ["**/*.md", "**/*.html"],
      "ignore": ["drafts/", "private.md", "**/generated/**"],
      "requireSourceUrl": false
    },
    "category": { "root": "docs", "maxDepth": 4, "prefix": "product" },
    "extract": { "minWords": 25 },
    "onDelete": "tombstone"
  },
  "metadata": { "defaults": { "product": "My product" }, "rules": [] }
}

Paths are relative to the manifest directory; . selects that directory. Absolute paths also work. include, exclude, and ignore accept arrays of paths or globs. A folder pattern such as drafts/ ignores its whole subtree. Nested manifests add exclusions relative to their own folder and can override metadata. Existing metadata-only manifests remain supported; crawl rules and transforms share this file.