GenAI Secret Sauce for Leveraging Cheap Code

Automate the Prep, Accelerate the Intelligence

Just today, I ran into a problem. I needed a script to take dozens of pages of documentation, stored as individual markdown files, and combine them into a single document. Why? Because I wanted to feed all that context to my GenAI chat, and manually selecting and uploading dozens of files is tedious and inefficient. My first thought was, surely someone has already written a script for this, I'll just check GitHub. But then I had an epiphany: in the time it would take me to search, evaluate, and potentially adapt an existing script, I could just ask an LLM to write exactly what I needed, tailored to my specific problem.

For a durable repository structure, learn how to use documentation stored as individual Markdown files.

This is where the power of GenAI comes in, fundamentally changing the game for tasks that involve processing and understanding large volumes of text. We're entering the era of "Cheap Code," where the effort required to generate code and automate tedious information handling is drastically reduced.

However, this "Cheap Code" revolution isn't automatically accessible to everyone. A significant hurdle is that:

• Problem Articulation: Many people struggle to articulate their specific problems in a way that a script can easily solve.
• Debugging Skills: Individuals may lack the technical skills needed to debug the code if the LLM's initial output isn't perfect.
• Awareness of Possibilities: Some simply don't realize that these kinds of creative, out-of-the-box solutions, such as generating a custom script for a unique data handling need, are even possible or beneficial.

Unlocking the full potential of "Cheap Code" often requires recognizing the opportunity and knowing how to translate a manual pain point into a solvable scripting task. This is one of the reasons I create genaisecretsauce.com, so I can share my thought process and hopefully expand your way of thinking to encompass these creative applications of GenAI.

The Documentation Combiner

Introducing Your LLM's New Best Friend

Ready for a peek behind the curtain? I'm sharing the source code for the script my LLM whipped up! This isn't just a simple file joiner, it's a smart Node.js utility that takes your scattered markdown files, merges them into one, keeps your original folder structure intact, and even builds a clickable table of contents. It's a prime example of "Cheap Code" in action, a custom, elegant solution generated fast by your AI copilot.

#!/usr/bin/env node

const fs = require('fs');
const path = require('path');

/**
 * Combines all markdown files in a directory into a single file
 * while preserving directory structure context
 */

// Parse command line arguments
const args = process.argv.slice(2);
if (args.length < 1) {
  console.error('Usage: node combineMarkdown.js <inputDirectory> [outputFile] [--exclude=pattern1,pattern2,...] [--title="Custom Title"]');
  console.error('Example: node combineMarkdown.js docs output.md --exclude=draft-*,temp/* --title="Project Documentation"');
  process.exit(1);
}

const inputDir = args[0];
let outputFile = 'combined_markdown.md';
let excludePatterns = [];
let documentTitle = 'Combined Markdown Files';

// Process remaining arguments
for (let i = 1; i < args.length; i++) {
  const arg = args[i];
  if (arg.startsWith('--exclude=')) {
    // Extract exclude patterns
    const patterns = arg.substring('--exclude='.length).split(',');
    excludePatterns = patterns.map(p => new RegExp(p.replace(/\*/g, '.*')));
  } else if (arg.startsWith('--title=')) {
    // Extract custom title
    documentTitle = arg.substring('--title='.length);
    // Remove surrounding quotes if present
    if ((documentTitle.startsWith('"') && documentTitle.endsWith('"')) || 
        (documentTitle.startsWith("'") && documentTitle.endsWith("'"))) {
      documentTitle = documentTitle.substring(1, documentTitle.length - 1);
    }
  } else if (!arg.startsWith('--')) {
    // Assume it's the output file
    outputFile = arg;
  }
}

// Validate input directory
if (!fs.existsSync(inputDir)) {
  console.error(`Error: Directory "${inputDir}" does not exist.`);
  process.exit(1);
}

// Function to check if a file should be excluded based on patterns
function shouldExclude(filePath, patterns) {
  if (patterns.length === 0) return false;
  
  return patterns.some(pattern => pattern.test(filePath));
}

// Function to find all markdown files in a directory and its subdirectories
function findMarkdownFiles(dir, fileList = [], baseDir = dir) {
  const files = fs.readdirSync(dir);
  
  files.forEach(file => {
    const filePath = path.join(dir, file);
    const relativePath = path.relative(baseDir, filePath);
    
    // Skip if the file or directory matches an exclude pattern
    if (shouldExclude(relativePath, excludePatterns)) {
      console.log(`Skipping excluded path: ${relativePath}`);
      return;
    }
    
    const stat = fs.statSync(filePath);
    
    if (stat.isDirectory()) {
      // Recursively search subdirectories
      findMarkdownFiles(filePath, fileList, baseDir);
    } else if (file.toLowerCase().endsWith('.md')) {
      // Add markdown files to the list with their relative path
      fileList.push({
        absolutePath: filePath,
        relativePath: relativePath
      });
    }
  });
  
  return fileList;
}

// Function to read file content
function readFileContent(filePath) {
  try {
    return fs.readFileSync(filePath, 'utf8');
  } catch (error) {
    console.error(`Error reading file ${filePath}: ${error.message}`);
    return `[Error reading file: ${error.message}]`;
  }
}

// Function to organize files by directory structure
function organizeByDirectory(files) {
  const structure = {};
  
  files.forEach(file => {
    const dirPath = path.dirname(file.relativePath);
    if (!structure[dirPath]) {
      structure[dirPath] = [];
    }
    structure[dirPath].push(file);
  });
  
  return structure;
}

// Function to generate output content with headers and table of contents
function generateOutput(structure) {
  let output = `# ${documentTitle}\n\nGenerated on ${new Date().toLocaleString()}\n\n`;
  let tableOfContents = `## Table of Contents\n\n`;
  let contentSection = '';
  
  // Sort directories to maintain consistent order
  const directories = Object.keys(structure).sort();
  
  // Generate table of contents
  directories.forEach(dir => {
    const dirName = dir === '.' ? 'Root Directory' : `Directory: ${dir}`;
    const dirLink = dir === '.' ? 'root-directory' : `directory-${dir.replace(/\//g, '-')}`;
    tableOfContents += `- [${dirName}](#${dirLink})\n`;
    
    // Add file links under each directory
    const files = structure[dir].sort((a, b) => 
      a.relativePath.localeCompare(b.relativePath)
    );
    
    files.forEach(file => {
      const fileName = path.basename(file.relativePath);
      const fileLink = `file-${file.relativePath.replace(/\//g, '-').replace(/\./g, '-')}`;
      tableOfContents += `  - [${fileName}](#${fileLink})\n`;
    });
  });
  
  tableOfContents += `\n---\n\n`;
  
  // Generate content sections
  directories.forEach(dir => {
    // Add directory header
    if (dir === '.') {
      contentSection += `<a id="root-directory"></a>\n## Root Directory\n\n`;
    } else {
      contentSection += `<a id="directory-${dir.replace(/\//g, '-')}"></a>\n## Directory: ${dir}\n\n`;
    }
    
    // Sort files within directory
    const files = structure[dir].sort((a, b) => 
      a.relativePath.localeCompare(b.relativePath)
    );
    
    // Add each file's content with a header
    files.forEach(file => {
      const fileName = path.basename(file.relativePath);
      const content = readFileContent(file.absolutePath);
      const fileLink = `file-${file.relativePath.replace(/\//g, '-').replace(/\./g, '-')}`;
      
      contentSection += `<a id="${fileLink}"></a>\n### File: ${file.relativePath}\n\n`;
      contentSection += `${content}\n\n`;
      contentSection += `---\n\n`; // Separator between files
    });
  });
  
  return output + tableOfContents + contentSection;
}

// Main execution
try {
  console.log(`Scanning directory: ${inputDir}`);
  const markdownFiles = findMarkdownFiles(inputDir);
  
  if (markdownFiles.length === 0) {
    console.log('No markdown files found.');
    process.exit(0);
  }
  
  console.log(`Found ${markdownFiles.length} markdown files.`);
  
  const fileStructure = organizeByDirectory(markdownFiles);
  const outputContent = generateOutput(fileStructure);
  
  // Write to output file
  fs.writeFileSync(outputFile, outputContent);
  console.log(`Successfully combined markdown files into: ${outputFile}`);
  
} catch (error) {
  console.error(`Error: ${error.message}`);
  process.exit(1);
}

Why do this?
Think about how you learn complex information. Do you prefer a scattered pile of loose pages, or a well organized book with chapters, sections, and an index? The answer is obvious. LLMs are similar. They thrive on structured, comprehensive input.

By consolidating your documentation into a single, easily digestible source, you're not just making data accessible, you're providing your GenAI agent with a dramatically enhanced information source that unlocks deeper understanding:

1. The Full Narrative: Instead of getting isolated snippets that might lack context, the LLM receives the complete story. This eliminates guesswork and helps it connect the dots between related concepts, code modules, or project specifications scattered across different files. It sees the forest, not just individual trees.
2. Implicit Hierarchy Made Explicit: The preserved directory structure, reflected in the combined document's organization and table of contents, gives the LLM a crucial understanding of how pieces of information relate hierarchically. Parent directories become main topics, subdirectories become sub topics (building a mental model for the AI).
3. Navigational Superpowers (for AI & Humans): A built in table of contents and clear section headers make it trivial for the LLM (and you!) to quickly jump to relevant sections, reference specific details, and synthesize information from disparate parts of your documentation. Tools like Google's Notebook LM, designed specifically for interacting with documents, unlock their full potential when given this kind of rich, navigable context.

Installation

1. Copy the script into a file named combineMarkdown.js
2. Make sure you have Node.js installed
3. Make the script executable:

chmod +x combineMarkdown.js

Usage

node combineMarkdown.js <inputDirectory> [outputFile]

If no output file is specified, the script defaults to combined_markdown.md in the current directory.

With options:

node combineMarkdown.js <inputDirectory> [outputFile] [--exclude=pattern1,pattern2,...] [--title="Custom Title"]

Examples

Combine all Markdown files in the 'docs' directory into 'combined.md':

node combineMarkdown.js docs combined.md

Combine files with a custom title:

node combineMarkdown.js docs combined.md --title="Project Documentation"

Exclude draft files and temporary directories:

node combineMarkdown.js docs combined.md --exclude=draft-*,temp/*

Combine with both custom title and exclusions:

node combineMarkdown.js docs combined.md --exclude=draft-*,temp/* --title="Project Documentation"

Once you have your master combined project docs.md file, a world of possibilities opens up:

• Upload it directly to context aware tools like Google's Notebook LM for deep dive analysis and Q&A.
• Include it seamlessly as part of the prompt or context window when interacting with coding assistants or other LLM agents.
• Use it as a single, definitive, and easily searchable reference source for any LLM based development or knowledge work task related to that project.

Note that writing and publishing this article took longer than writing the script.