September 8, 2025

CI Configuration Translation

A compact research companion for translating Travis CI configurations into GitHub Actions, including the guideline-enhanced prompt, evaluation results, iterative refinement behavior, and dataset access.

Best BSR

88.5%

Guideline + IR

Best LMR

96.6%

Guideline + IR

Repositories

200

Public benchmark dataset

Prompt Design

1. Guidelines

#

Below is the guideline-enhanced prompt used in the article. It focuses on syntax, platform discrepancies, environment setup, and workflow logic.

guidelines.yaml
- Syntax:
    - Ensure YAML file follows proper syntax, including required symbols, e.g., colons, hyphens, or quotation marks.
    - Verify that GitHub Actions YAML file follows proper indentation and structural conventions.
    - Ensure all used definitions are correctly defined and properly placed.
    - Ensure the value types match their expected format when such fields exist in the source configuration, e.g., the setup-python action expects a single version value, not a list.

- Platform Discrepancy:
    - Translate Travis CI keys only when a clear, direct GitHub Actions equivalent exists. Skip any Travis CI-specific or unsupported keys.
    - Verify that all expressions are properly converted to GitHub Actions syntax, e.g., 'env: TOX_ENV=flake8' to 'TOX_ENV: flake8'.
    - Retain only x64 and arm64 if architectures are explicitly defined; remove all other architectures.
    - Ensure all version numbers are in string format, e.g., Python '3.10'.
    - Explicitly configure fields that are implicitly defaulted in Travis CI as equivalent GitHub Actions steps, e.g., actions/checkout@v4 and language runtime setup.
    - Ensure all packages are properly installed and configured when present in the Travis CI configuration, e.g., migrate tox-travis to tox-gh-actions if applicable.

- Environment:
    - Ensure all referenced GitHub Actions currently exist; avoid non-existent actions.
    - Ensure all required tokens are properly configured for the migrated steps, e.g., set GITHUB_TOKEN in the coveralls step if such a step exists.

- Logic:
    - Ensure all trigger events are correctly translated, e.g., avoid using tags as the sole trigger event unless the Travis CI configuration explicitly relies on tag-only builds.
    - Ensure all execution orders are correctly translated, e.g., deploy job should be configured with the needs parameter to execute after test job.
    - Ensure conditional statements are correctly translated.
    - Do not introduce extra jobs, steps, operating systems, or matrix dimensions beyond those implied by the Travis CI configuration, except where required to faithfully reproduce Travis CI default behavior.
    - Ensure that all transferable steps have been migrated.

Evaluation

2. Results

#
Metric Importer Basic One-shot Guideline IR Guideline + IR
BSR 31.5% 63.0% 57.0% 71.5% 85.5% 88.5%
LMR 95.2% 96.0% 95.6% 95.8% 96.5% 96.6%

Ablation

3. Iterative Refinement

#
Strategies 0 1 2 3 4 5
IR 126 155 164 167 171 171
Guideline + IR 143 165 170 176 177 177

Examples

Examples of Equivalent Travis CI and GitHub Actions Configuration

#
Example of equivalent Travis CI and GitHub Actions configuration
Example of equivalent Travis CI and GitHub Actions configuration

Resources

4. Data Set

The dataset comprises 200 GitHub repositories. The source code is publicly available through the anonymized repository below.

Visit citranslation