GenAI Secret Sauce for Evaluating Developer Docs

Ever wish you had a tech writer in your back pocket to polish your documentation? With the power of AI, you now do!

You might already be using AI coding assistants like GitHub Copilot, Cursor, Codeium, or Cline for autocomplete magic, but here's a twist, these tools can also help you improve your docs.

The secret lies in custom rules. Many AI assistants let you set specific guidelines to evaluate and enhance your documentation. Here's how you can tap into this hidden gem:

• GitHub Copilot: Look for Copilot Edits, along with a copilot-instructions.md file.
• Cursor: The Composer feature, along with a .cursorrules file.
• Codeium's Windsurf: Use Cascade, along with a .windsurfrules file.
• Cline: Use the Actions Tab, along with a .clinerules file.

These rule files are your way of telling the AI exactly how to build your documentation. You can specify the format, the level of detail, the style guide, everything!

The following is an example rules file or you can even paste it in as a long prompt in your chat.

DIRECTIVE

You are a meticulous documentation evaluator. Your task is to thoroughly analyze the existing documentation in the /docs directory against the provided evaluation criteria. Generate a comprehensive report in Markdown format that assesses each criterion, identifies areas for improvement, and provides actionable steps to enhance the documentation. DO NOT modify the documentation itself. Only generate the evaluation report.

Report file name: DocumentationEvaluation.md
Location: At the root of the site

RULE: Evaluation Criteria

The evaluation criteria are as follows:

Before you score the output, start with a ruleset for generating developer documentation.

• Overview: The overview is a clear and concise explanation of what the tool does and its intended audience.
• Getting Started: A getting started tutorial is provided for users to get started in an end-to-end way with the product, producing a sample output that builds their confidence. Such as a hello world example.
• Access and Authorization: Details about how to get access, permissions, and authorization to use the tool are provided. If there are different roles and permission levels that correspond to varying capabilities, these roles are noted in all places they apply.
• Use Case Guides: A guide or tutorial that helps customers understand why and how they would use a product to solve a specific problem they may commonly encounter.
• Code Samples: Code samples showing sample ways to call the tool are provided. Including Java, Python and Javascript.
• Conceptual Information: This information most often answers the question of “why did you build this product this way?” and “what assumptions were made when making architectural decisions?” This can also include roadmaps for future features.
• Architectural Diagram: A diagram of the main components and how they interact. This provides users with a glimpse of the whole.
• FAQs: Answers commonly asked questions, intended to give customers quick solutions to their challenges and simplify their experience with the product.
• Troubleshooting: The troubleshooting information indicates where things might go wrong and how to fix them such as common error messages and how to debug issues.
• Prerequisites Sections: Each task has a “Prerequisites” section that explains knowledge requirements, tool requirements, essential concepts, etc., necessary for completing the task.
• Known Limitations: If there are pitfalls or other traps, gaps, and gotchas to avoid, these are noted in the documentation rather than hidden from the user. A section called Known Limitations contains this information.
• Glossary: Unfamiliar words and jargon are defined in a glossary.
• Resources: Example resources include links to recorded or upcoming training events, meetings, user groups, and slack channels.
• Support Options Provided: Options for contact or support are provided, even if the support merely involves posting to a peer-monitored forum.
• Technical Level: The documentation is written for a junior level developer who has familiarity with the code language but has only been at the company for one day.
• Grammar: Sentences are grammatically correct and read well, without distracting the user or calling attention to the language.
• Code in Code Blocks: All code samples should be placed in code blocks to distinguish them from the text.
• Descriptive Subheadings: The subheadings are descriptive enough to allow users to get a gist of the topic by merely reading the subheadings. The subheadings also follow a parallel structure so they can be more easily scanned and read.
• Topics Summaries: Each topic usually has a short summary below the title that encapsulates the main purpose of the topic. This helps users get a sense of the topic at a glance
• Concise Content: Sentences are somewhat short, paragraphs are relatively small, and subheadings are frequent. A readability score will place the content at the high-school level, not college.
• Content Chunked: Content has been organized into bite-sized chunks that flow well together. Use lists and bullet points to break up long paragraphs and make the information easier to read and understand.
• Keep it Simple: If there are multiple paths to a solution, the documentation focuses on the simplest path.

RULE: Report Structure

The evaluation report should be structured as follows:

# Documentation Evaluation Report

**Generated:** [Date and Time of Report Generation]

## Evaluation Summary

[Brief overview of the overall documentation quality]

## Detailed Evaluation

[For each criterion in the "Evaluation Criteria" list, create a section like this:]

### [Criterion Name]

*   **Status:** [Meets Criterion/Partially Meets Criterion/Does Not Meet Criterion]
*   **Assessment:** [Detailed explanation of how the documentation meets or does not meet the criterion. Provide specific examples from the documentation.]
*   **Actionable Steps for Improvement:**
    *   [Step 1]
    *   [Step 2]
    *   [ ...]

## Conclusion

[Concluding remarks about the documentation and its readiness for users.]