scholar-evaluation

What this skill does

# Scholar Evaluation

## Overview

Apply the ScholarEval framework to systematically evaluate scholarly and research work. This skill provides structured evaluation methodology based on peer-reviewed research assessment criteria, enabling comprehensive analysis of academic papers, research proposals, literature reviews, and scholarly writing across multiple quality dimensions.

## When to Use This Skill

Use this skill when:
- Evaluating research papers for quality and rigor
- Assessing literature review comprehensiveness and quality
- Reviewing research methodology design
- Scoring data analysis approaches
- Evaluating scholarly writing and presentation
- Providing structured feedback on academic work
- Benchmarking research quality against established criteria
- Assessing publication readiness for target venues
- Providing quantitative evaluation to complement qualitative peer review

## Visual Enhancement with Scientific Schematics

**When creating documents with this skill, always consider adding scientific diagrams and schematics to enhance visual communication.**

If your document does not already contain schematics or diagrams:
- Use the **scientific-schematics** skill to generate AI-powered publication-quality diagrams
- Simply describe your desired diagram in natural language
- Nano Banana Pro will automatically generate, review, and refine the schematic

**For new documents:** Scientific schematics should be generated by default to visually represent key concepts, workflows, architectures, or relationships described in the text.

**How to generate schematics:**
```bash
python scripts/generate_schematic.py "your diagram description" -o figures/output.png
```

The AI will automatically:
- Create publication-quality images with proper formatting
- Review and refine through multiple iterations
- Ensure accessibility (colorblind-friendly, high contrast)
- Save outputs in the figures/ directory

**When to add schematics:**
- Evaluation framework diagrams
- Quality assessment criteria decision trees
- Scholarly workflow visualizations
- Assessment methodology flowcharts
- Scoring rubric visualizations
- Evaluation process diagrams
- Any complex concept that benefits from visualization

For detailed guidance on creating schematics, refer to the scientific-schematics skill documentation.

---

## Evaluation Workflow

### Step 1: Initial Assessment and Scope Definition

Begin by identifying the type of scholarly work being evaluated and the evaluation scope:

**Work Types:**
- Full research paper (empirical, theoretical, or review)
- Research proposal or protocol
- Literature review (systematic, narrative, or scoping)
- Thesis or dissertation chapter
- Conference abstract or short paper

**Evaluation Scope:**
- Comprehensive (all dimensions)
- Targeted (specific aspects like methodology or writing)
- Comparative (benchmarking against other work)

Ask the user to clarify if the scope is ambiguous.

### Step 2: Dimension-Based Evaluation

Systematically evaluate the work across the ScholarEval dimensions. For each applicable dimension, assess quality, identify strengths and weaknesses, and provide scores where appropriate.

Refer to `references/evaluation_framework.md` for detailed criteria and rubrics for each dimension.

**Core Evaluation Dimensions:**

1. **Problem Formulation & Research Questions**
   - Clarity and specificity of research questions
   - Theoretical or practical significance
   - Feasibility and scope appropriateness
   - Novelty and contribution potential

2. **Literature Review**
   - Comprehensiveness of coverage
   - Critical synthesis vs. mere summarization
   - Identification of research gaps
   - Currency and relevance of sources
   - Proper contextualization

3. **Methodology & Research Design**
   - Appropriateness for research questions
   - Rigor and validity
   - Reproducibility and transparency
   - Ethical considerations
   - Limitations acknowledgment

4. **Data Collection & Sources**
   - Quality and appropriateness of data
   - Sample size and representativeness
   - Data collection procedures
   - Source credibility and reliability

5. **Analysis & Interpretation**
   - Appropriateness of analytical methods
   - Rigor of analysis
   - Logical coherence
   - Alternative explanations considered
   - Results-claims alignment

6. **Results & Findings**
   - Clarity of presentation
   - Statistical or qualitative rigor
   - Visualization quality
   - Interpretation accuracy
   - Implications discussion

7. **Scholarly Writing & Presentation**
   - Clarity and organization
   - Academic tone and style
   - Grammar and mechanics
   - Logical flow
   - Accessibility to target audience

8. **Citations & References**
   - Citation completeness
   - Source quality and appropriateness
   - Citation accuracy
   - Balance of perspectives
   - Adherence to citation standards

### Step 3: Scoring and Rating

For each evaluated dimension, provide:

**Qualitative Assessment:**
- Key strengths (2-3 specific points)
- Areas for improvement (2-3 specific points)
- Critical issues (if any)

**Quantitative Scoring (Optional):**
Use a 5-point scale where applicable:
- 5: Excellent - Exemplary quality, publishable in top venues
- 4: Good - Strong quality with minor improvements needed
- 3: Adequate - Acceptable quality with notable areas for improvement
- 2: Needs Improvement - Significant revisions required
- 1: Poor - Fundamental issues requiring major revision

To calculate aggregate scores programmatically, use `scripts/calculate_scores.py`.

### Step 4: Synthesize Overall Assessment

Provide an integrated evaluation summary:

1. **Overall Quality Assessment** - Holistic judgment of the work's scholarly merit
2. **Major Strengths** - 3-5 key strengths across dimensions
3. **Critical Weaknesses** - 3-5 primary areas requiring attention
4. **Priority Recommendations** - Ranked list of improvements by impact
5. **Publication Readiness** (if applicable) - Assessment of suitability for target venues

### Step 5: Provide Actionable Feedback

Transform evaluation findings into constructive, actionable feedback:

**Feedback Structure:**
- **Specific** - Reference exact sections, paragraphs, or page numbers
- **Actionable** - Provide concrete suggestions for improvement
- **Prioritized** - Rank recommendations by importance and feasibility
- **Balanced** - Acknowledge strengths while addressing weaknesses
- **Evidence-based** - Ground feedback in evaluation criteria

**Feedback Format Options:**
- Structured report with dimension-by-dimension analysis
- Annotated comments mapped to specific document sections
- Executive summary with key findings and recommendations
- Comparative analysis against benchmark standards

### Step 6: Contextual Considerations

Adjust evaluation approach based on:

**Stage of Development:**
- Early draft: Focus on conceptual and structural issues
- Advanced draft: Focus on refinement and polish
- Final submission: Comprehensive quality check

**Purpose and Venue:**
- Journal article: High standards for rigor and contribution
- Conference paper: Balance novelty with presentation clarity
- Student work: Educational feedback with developmental focus
- Grant proposal: Emphasis on feasibility and impact

**Discipline-Specific Norms:**
- STEM fields: Emphasis on reproducibility and statistical rigor
- Social sciences: Balance quantitative and qualitative standards
- Humanities: Focus on argumentation and scholarly interpretation

## Resources

### references/evaluation_framework.md

Detailed evaluation criteria, rubrics, and quality indicators for each ScholarEval dimension. Load this reference when conducting evaluations to access specific assessment guidelines and scoring rubrics.

Search patterns for quick access:
- "Problem Formulation criteria"
- "Literature Review rubric"
- "Methodology assessment"
- "Data quality indicators"
- "Analysis rigor standards"
- "Writing quality checklist"

### scripts/calculate_scores.py