make thoughts searchable (#235)

* removed thoughts

* feat(thoughts): improve search functionality with searchable directory

- Remove automatic addition of thoughts/ to .gitignore
- Create searchable/ directory with hard links for better search support
- Update sync command to create hard links following symlinks
- Exclude CLAUDE.md from searchable directory to avoid duplicates
- Update documentation to explain searchable directory usage

The thoughts directory is now searchable by AI agents while maintaining
symlink structure for file operations like mv and rm.

* docs: update documentation for searchable thoughts directory

- Add searchable directory structure to THOUGHTS.md
- Explain how searchable directory works with hard links
- Remove references to automatic gitignore addition
- Add FAQ entries about searchable directory
- Add thoughts commands to README.md
- Link to detailed thoughts documentation

* formatting

* formatting

.gitignore

@@ -178,6 +178,3 @@ cython_debug/
 .Trashes
 
 .idea/
-
-# HumanLayer thoughts directory
-/thoughts/

hlyr/README.md

@@ -7,6 +7,7 @@ A unified CLI tool that provides:
 - Direct human contact from terminal or scripts
 - MCP (Model Context Protocol) server functionality
 - Integration with Claude Code SDK for approval workflows
+- Thoughts management system for developer notes and documentation
 
 ## Quickstart
 
@@ -184,6 +185,39 @@ humanlayer mcp <subcommand>
 - `wrapper` - Wrap an existing MCP server with human approval functionality (not implemented yet)
 - `inspector [command]` - Run MCP inspector for debugging MCP servers (defaults to 'serve')
 
+### `thoughts`
+
+Manage developer thoughts and notes separately from code repositories.
+
+```bash
+humanlayer thoughts <subcommand>
+```
+
+**Subcommands:**
+
+- `init` - Initialize thoughts for the current repository
+- `sync` - Manually sync thoughts and update searchable index
+- `status` - Check the status of your thoughts setup
+- `config` - View or edit thoughts configuration
+
+**Examples:**
+
+```bash
+# Initialize thoughts for a new project
+humanlayer thoughts init
+
+# Sync thoughts after making changes
+humanlayer thoughts sync -m "Updated architecture notes"
+
+# Check status
+humanlayer thoughts status
+
+# View configuration
+humanlayer thoughts config --json
+```
+
+The thoughts system keeps your notes separate from code while making them easily accessible to AI assistants. See the [Thoughts documentation](./THOUGHTS.md) for detailed information.
+
 ## Use Cases
 
 - **CI/CD Pipelines**: Get human approval before deploying

hlyr/THOUGHTS.md

@@ -45,8 +45,12 @@ your-project/
 │   ├── global/           # → ~/thoughts/global
 │   │   ├── alice/        # Your cross-repo notes
 │   │   └── shared/       # Team cross-repo notes
+│   ├── searchable/       # Hard links for AI search (auto-generated)
+│   │   ├── alice/        # Hard links to alice's files
+│   │   ├── shared/       # Hard links to shared files
+│   │   └── global/       # Hard links to global files
 │   └── CLAUDE.md         # Auto-generated context for AI
-└── .gitignore            # Updated to exclude thoughts/
+└── .gitignore
 ```
 
 Your central thoughts repository:
@@ -70,10 +74,19 @@ Your central thoughts repository:
 The system automatically syncs your thoughts when you commit code:
 
 1. **Pre-commit hook** - Prevents thoughts/ from being committed to your code repo
-2. **Post-commit hook** - Syncs thoughts changes to your thoughts repository
+2. **Post-commit hook** - Syncs thoughts changes to your thoughts repository and updates the searchable directory
 
 This means you can work naturally - edit thoughts alongside code, and they'll be kept in sync automatically.
 
+### Searchable Directory
+
+The `thoughts/searchable/` directory contains read-only hard links to all thoughts files. This allows AI tools to search your thoughts content without needing to follow symlinks. The searchable directory:
+
+- Is automatically updated when you run `humanlayer thoughts sync`
+- Contains hard links (not copies) to preserve disk space
+- Is read-only to prevent accidental edits
+- Should not be edited directly - always edit the original files
+
 ## Commands
 
 ### `humanlayer thoughts init`
@@ -245,7 +258,7 @@ humanlayer thoughts init
 
 ### CI/CD Integration
 
-The thoughts directory is automatically ignored by git, so it won't affect CI/CD pipelines. The symlinks are also excluded, ensuring clean builds.
+The thoughts directory is protected by a pre-commit hook that prevents accidental commits to your code repository. This ensures clean CI/CD pipelines while keeping thoughts accessible for searching and development.
 
 ## Privacy & Security
 
@@ -275,6 +288,12 @@ A: Currently all projects share the same thoughts repo, but use different subdir
 **Q: Why can't I use "global" as my username?**
 A: "global" is reserved for cross-project thoughts. This ensures the directory structure remains clear.
 
+**Q: Why do I need a searchable directory?**
+A: Many search tools don't follow symlinks by default. The searchable directory contains hard links to all your thoughts files, making them easily searchable by AI assistants and other tools.
+
+**Q: Can I edit files in the searchable directory?**
+A: No, files in searchable/ are read-only. Always edit the original files (e.g., edit thoughts/alice/todo.md, not thoughts/searchable/alice/todo.md).
+
 ## Contributing
 
 The thoughts system is part of HumanLayer. To contribute:

hlyr/src/commands/thoughts/init.ts

@@ -137,6 +137,21 @@ It is managed by the HumanLayer thoughts system and should not be committed to t
 - \`global/\` → Cross-repository thoughts (symlink to ${globalPath})
   - \`${user}/\` - Your personal notes that apply across all repositories
   - \`shared/\` - Team-shared notes that apply across all repositories
+- \`searchable/\` → Read-only hard links for searching (auto-generated)
+
+## Searching in Thoughts
+
+The \`searchable/\` directory contains read-only hard links to all thoughts files accessible in this repository. This allows search tools to find content without following symlinks.
+
+**IMPORTANT**:
+- Files found in \`thoughts/searchable/\` are read-only copies
+- To edit any file, use the original path (e.g., edit \`thoughts/${user}/todo.md\`, not \`thoughts/searchable/${user}/todo.md\`)
+- The \`searchable/\` directory is automatically updated when you run \`humanlayer thoughts sync\`
+
+This design ensures that:
+1. Search tools can find all your thoughts content easily
+2. The symlink structure remains intact for git operations
+3. You can't accidentally edit the wrong copy of a file
 
 ## Usage
 
@@ -446,6 +461,20 @@ export async function thoughtsInitCommand(options: InitOptions): Promise<void> {
     // Create thoughts directory in current repo
     const thoughtsDir = path.join(currentRepo, 'thoughts')
     if (fs.existsSync(thoughtsDir)) {
+      // Handle searchable directories specially if they exist (might have read-only permissions)
+      const searchableDir = path.join(thoughtsDir, 'searchable')
+      const oldSearchDir = path.join(thoughtsDir, '.search')
+
+      for (const dir of [searchableDir, oldSearchDir]) {
+        if (fs.existsSync(dir)) {
+          try {
+            // Reset permissions so we can delete it
+            execSync(`chmod -R 755 "${dir}"`, { stdio: 'pipe' })
+          } catch {
+            // Ignore chmod errors
+          }
+        }
+      }
       fs.rmSync(thoughtsDir, { recursive: true, force: true })
     }
     fs.mkdirSync(thoughtsDir)
@@ -481,20 +510,6 @@ export async function thoughtsInitCommand(options: InitOptions): Promise<void> {
     // Setup git hooks
     setupGitHooks(currentRepo)
 
-    // Add thoughts to .gitignore if not already there
-    const gitignorePath = path.join(currentRepo, '.gitignore')
-    let gitignoreContent = ''
-
-    if (fs.existsSync(gitignorePath)) {
-      gitignoreContent = fs.readFileSync(gitignorePath, 'utf8')
-    }
-
-    if (!gitignoreContent.includes('/thoughts/') && !gitignoreContent.includes('thoughts/')) {
-      gitignoreContent += '\n# HumanLayer thoughts directory (root level only)\n/thoughts/\n'
-      fs.writeFileSync(gitignorePath, gitignoreContent)
-      console.log(chalk.green('✅ Added /thoughts/ to .gitignore'))
-    }
-
     console.log(chalk.green('✅ Thoughts setup complete!'))
     console.log('')
     console.log(chalk.blue('=== Summary ==='))
@@ -517,14 +532,14 @@ export async function thoughtsInitCommand(options: InitOptions): Promise<void> {
     console.log('Protection enabled:')
     console.log(`  ${chalk.green('✓')} Pre-commit hook: Prevents committing thoughts/`)
     console.log(`  ${chalk.green('✓')} Post-commit hook: Auto-syncs thoughts after commits`)
-    console.log(`  ${chalk.green('✓')} Added to .gitignore`)
     console.log('')
     console.log('Next steps:')
+    console.log(`  1. Run ${chalk.cyan('humanlayer thoughts sync')} to create the searchable index`)
     console.log(
-      `  1. Create markdown files in ${chalk.cyan(`thoughts/${config.user}/`)} for your notes`,
+      `  2. Create markdown files in ${chalk.cyan(`thoughts/${config.user}/`)} for your notes`,
     )
-    console.log(`  2. Your thoughts will sync automatically when you commit code`)
-    console.log(`  3. Run ${chalk.cyan('humanlayer thoughts status')} to check sync status`)
+    console.log(`  3. Your thoughts will sync automatically when you commit code`)
+    console.log(`  4. Run ${chalk.cyan('humanlayer thoughts status')} to check sync status`)
   } catch (error) {
     console.error(chalk.red(`Error during thoughts init: ${error}`))
     process.exit(1)

hlyr/src/commands/thoughts/sync.ts

@@ -70,6 +70,115 @@ function syncThoughts(thoughtsRepo: string, message: string): void {
   }
 }
 
+function createSearchDirectory(thoughtsDir: string): void {
+  const searchDir = path.join(thoughtsDir, 'searchable')
+  const oldSearchDir = path.join(thoughtsDir, '.search')
+
+  // Remove old .search directory if it exists
+  if (fs.existsSync(oldSearchDir)) {
+    try {
+      execSync(`chmod -R 755 "${oldSearchDir}"`, { stdio: 'pipe' })
+    } catch {
+      // Ignore chmod errors
+    }
+    fs.rmSync(oldSearchDir, { recursive: true, force: true })
+  }
+
+  // Remove existing searchable directory if it exists
+  if (fs.existsSync(searchDir)) {
+    try {
+      // Reset permissions so we can delete it
+      execSync(`chmod -R 755 "${searchDir}"`, { stdio: 'pipe' })
+    } catch {
+      // Ignore chmod errors
+    }
+    fs.rmSync(searchDir, { recursive: true, force: true })
+  }
+
+  // Create new .search directory
+  fs.mkdirSync(searchDir, { recursive: true })
+
+  // Function to recursively find all files through symlinks
+  function findFilesFollowingSymlinks(
+    dir: string,
+    baseDir: string = dir,
+    visited: Set<string> = new Set(),
+  ): string[] {
+    const files: string[] = []
+
+    // Resolve symlinks to avoid cycles
+    const realPath = fs.realpathSync(dir)
+    if (visited.has(realPath)) {
+      return files
+    }
+    visited.add(realPath)
+
+    const entries = fs.readdirSync(dir, { withFileTypes: true })
+
+    for (const entry of entries) {
+      const fullPath = path.join(dir, entry.name)
+      if (entry.isDirectory() && !entry.name.startsWith('.')) {
+        files.push(...findFilesFollowingSymlinks(fullPath, baseDir, visited))
+      } else if (entry.isSymbolicLink() && !entry.name.startsWith('.')) {
+        try {
+          const stat = fs.statSync(fullPath)
+          if (stat.isDirectory()) {
+            files.push(...findFilesFollowingSymlinks(fullPath, baseDir, visited))
+          } else if (stat.isFile() && path.basename(fullPath) !== 'CLAUDE.md') {
+            files.push(path.relative(baseDir, fullPath))
+          }
+        } catch {
+          // Ignore broken symlinks
+        }
+      } else if (entry.isFile() && !entry.name.startsWith('.') && entry.name !== 'CLAUDE.md') {
+        files.push(path.relative(baseDir, fullPath))
+      }
+    }
+
+    return files
+  }
+
+  // Get all files accessible through the thoughts directory (following symlinks)
+  const allFiles = findFilesFollowingSymlinks(thoughtsDir)
+
+  // Create hard links in .search directory
+  let linkedCount = 0
+  for (const relPath of allFiles) {
+    const sourcePath = path.join(thoughtsDir, relPath)
+    const targetPath = path.join(searchDir, relPath)
+
+    // Create directory structure
+    const targetDir = path.dirname(targetPath)
+    if (!fs.existsSync(targetDir)) {
+      fs.mkdirSync(targetDir, { recursive: true })
+    }
+
+    try {
+      // Resolve symlink to get the real file path
+      const realSourcePath = fs.realpathSync(sourcePath)
+      // Create hard link to the real file
+      fs.linkSync(realSourcePath, targetPath)
+      linkedCount++
+    } catch {
+      // Silently skip files we can't link (e.g., different filesystems)
+    }
+  }
+
+  // Make .search directory read-only
+  try {
+    // First set directories to be readable and traversable
+    execSync(`find "${searchDir}" -type d -exec chmod 755 {} +`, { stdio: 'pipe' })
+    // Then set files to be read-only
+    execSync(`find "${searchDir}" -type f -exec chmod 444 {} +`, { stdio: 'pipe' })
+    // Finally make directories read-only but still traversable
+    execSync(`find "${searchDir}" -type d -exec chmod 555 {} +`, { stdio: 'pipe' })
+  } catch {
+    // Ignore chmod errors on systems that don't support it
+  }
+
+  console.log(chalk.gray(`Created ${linkedCount} hard links in searchable directory`))
+}
+
 export async function thoughtsSyncCommand(options: SyncOptions): Promise<void> {
   try {
     // Check if thoughts are configured
@@ -107,6 +216,10 @@ export async function thoughtsSyncCommand(options: SyncOptions): Promise<void> {
       }
     }
 
+    // Create .search directory with hard links
+    console.log(chalk.blue('Creating searchable index...'))
+    createSearchDirectory(thoughtsDir)
+
     // Sync the thoughts repository
     console.log(chalk.blue('Syncing thoughts...'))
     syncThoughts(config.thoughtsRepo, options.message || '')