Prompt Design
1. Guidelines
Below is the guideline-enhanced prompt used in the article. It focuses on syntax, platform discrepancies, environment setup, and workflow logic.
- Syntax:
- Ensure YAML file follows proper syntax, including required symbols, e.g., colons, hyphens, or quotation marks.
- Verify that GitHub Actions YAML file follows proper indentation and structural conventions.
- Ensure all used definitions are correctly defined and properly placed.
- Ensure the value types match their expected format when such fields exist in the source configuration, e.g., the setup-python action expects a single version value, not a list.
- Platform Discrepancy:
- Translate Travis CI keys only when a clear, direct GitHub Actions equivalent exists. Skip any Travis CI-specific or unsupported keys.
- Verify that all expressions are properly converted to GitHub Actions syntax, e.g., 'env: TOX_ENV=flake8' to 'TOX_ENV: flake8'.
- Retain only x64 and arm64 if architectures are explicitly defined; remove all other architectures.
- Ensure all version numbers are in string format, e.g., Python '3.10'.
- Explicitly configure fields that are implicitly defaulted in Travis CI as equivalent GitHub Actions steps, e.g., actions/checkout@v4 and language runtime setup.
- Ensure all packages are properly installed and configured when present in the Travis CI configuration, e.g., migrate tox-travis to tox-gh-actions if applicable.
- Environment:
- Ensure all referenced GitHub Actions currently exist; avoid non-existent actions.
- Ensure all required tokens are properly configured for the migrated steps, e.g., set GITHUB_TOKEN in the coveralls step if such a step exists.
- Logic:
- Ensure all trigger events are correctly translated, e.g., avoid using tags as the sole trigger event unless the Travis CI configuration explicitly relies on tag-only builds.
- Ensure all execution orders are correctly translated, e.g., deploy job should be configured with the needs parameter to execute after test job.
- Ensure conditional statements are correctly translated.
- Do not introduce extra jobs, steps, operating systems, or matrix dimensions beyond those implied by the Travis CI configuration, except where required to faithfully reproduce Travis CI default behavior.
- Ensure that all transferable steps have been migrated.
Evaluation
2. Results
| Metric | Importer | Basic | One-shot | Guideline | IR | Guideline + IR |
|---|---|---|---|---|---|---|
| BSR | 31.5% | 63.0% | 57.0% | 71.5% | 85.5% | 88.5% |
| LMR | 95.2% | 96.0% | 95.6% | 95.8% | 96.5% | 96.6% |
Ablation
3. Iterative Refinement
| Strategies | 0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
| IR | 126 | 155 | 164 | 167 | 171 | 171 |
| Guideline + IR | 143 | 165 | 170 | 176 | 177 | 177 |
Examples
Examples of Equivalent Travis CI and GitHub Actions Configuration
Resources
4. Data Set
The dataset comprises 200 GitHub repositories. The source code is publicly available through the anonymized repository below.
Visit citranslation