Redesign framework lesson test file recreation architecture

[meanmail]

Overview

Recent anti-cheating implementation recreates test files before Check and during framework lesson navigation. However, the architecture has issues: code duplication, unclear boundaries between student/course files, and misunderstanding about test file specification.

Related Commit

cafd1a5534 - Fix framework lesson persistence, anti-cheating, and file handling

Key Understanding: Test Files are Defined by Testing Libraries

Important: Test file paths and naming conventions are defined by the testing libraries (hs-test for Java, hs-test-python, etc.), NOT by the plugin configurators. The configurators must follow the testing library specifications.

Example testing libraries:

• Java/Kotlin: https://github.com/hyperskill/hs-test - defines test structure
• Python: hs-test-python - defines where test files should be
• JavaScript: hs-test-js - etc.

The plugin's testDirs configuration (e.g., ["test"] for Python, ["hstest", "test"] for JavaScript) must match what the testing libraries expect, not the other way around.

Problem Areas

1. Code Duplication

Test file recreation logic appears in multiple places:

• intellij-plugin/hs-core/src/org/hyperskill/academy/learning/actions/CheckAction.kt:167-194
• intellij-plugin/hs-core/src/org/hyperskill/academy/learning/framework/impl/FrameworkLessonManagerImpl.kt:219-245

2. Test File Detection Logic

Current implementation in HyperskillConfigurator.kt:56-61:

override fun isTestFile(task: Task, path: String): Boolean {
    // Use this.testDirs instead of delegating to baseConfigurator
    val isTestFile = path == testFileName || testDirs.any { testDir -> VfsUtilCore.isEqualOrAncestor(testDir, path) }
    val taskFile = task.getTaskFile(path)
    return isTestFile || taskFile?.isVisible == false
}

This logic must correctly implement the testing library specifications for each language:

• Python: testDirs = ["test"] (recent fix in e66f7a1c07)
• JavaScript: testDirs = ["hstest", "test"]
• Java/Kotlin: based on hs-test requirements
• Go, Rust, C++, etc.: each has its own testing library conventions

3. Unclear File Classification

Recent change from isVisible to isLearnerCreated filter may have side effects. Need clear separation between:

• Student-created files (should be preserved)
• Course-provided immutable files (should be recreated from course definition)
• Test files (defined by testing library, should be recreated to prevent cheating)

4. Performance Concerns

VFS operations are expensive. Current implementation may recreate files unnecessarily.

What Makes This Hard

1. Testing library specifications: Must understand and follow conventions from multiple testing libraries (hs-test, hs-test-python, etc.)
2. Language-agnostic design: Must work correctly for 13 languages, each with different testing library conventions
3. Multiple trigger points: Recreation needed at Check, framework navigation, potentially task updates
4. Performance: Must avoid unnecessary file recreation (VFS writes are slow)
5. Edge cases:
   • Binary test files
   • Test files with placeholders
   • Deleted test files
   • Test files in non-standard locations (if testing library supports it)
   • Invisible files (isVisible == false)
6. Framework storage interaction: Must not interfere with student progress tracking
7. Anti-cheating security: Students must not be able to bypass checks by modifying test files

Requirements

• Research testing library specifications for each supported language (Java, Python, Kotlin, JavaScript, Go, Rust, C++, PHP, Scala, Shell, SQL, C#)
• Document how each testing library defines test file locations
• Design a centralized test file recreation service
• Implement proper abstraction for "immutable course files" vs "student files"
• Ensure testDirs configuration in each language configurator matches testing library requirements
• Handle all edge cases documented above
• Add intelligent caching to avoid redundant recreation
• Add comprehensive tests for anti-cheating scenarios:
  • Student modifies test file → Check → test file restored
  • Binary test files handled correctly
  • Invisible files handled correctly
  • Performance test with 50+ files
• No noticeable delay when navigating or checking (< 100ms for typical task)

Success Criteria

• Single source of truth for test file recreation logic
• Clear separation of concerns (file classification, recreation, storage)
• All languages work correctly (test against at least 3-4 different languages)
• Test file detection correctly follows testing library specifications
• Performance regression test passes
• Anti-cheating mechanism cannot be bypassed
• Documentation of how each testing library defines test files

Estimated Time

6-7 hours (includes research of testing libraries, design, implementation, testing across multiple languages)

References

• hs-test for Java: https://github.com/hyperskill/hs-test
• Recent fix for Python testDirs: commit e66f7a1c07
• HyperskillConfigurator constant: HYPERSKILL_TEST_DIR = "hstest"