Skip to content

feat() add crlf support for csv exports #2783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

7amza79
Copy link

@7amza79 7amza79 commented Jun 18, 2025

Add CRLF Support for CSV Exports

Description

This PR adds support for Windows-style line endings (CRLF) in CSV exports, while maintaining the default UNIX-style line endings (LF). This enhancement improves compatibility with Windows-based systems and tools that expect CRLF line endings.

Changes

  • Added line_ending configuration option to CSV writers with two possible values:
    • lf (default): Uses UNIX-style line endings
    • crlf: Uses Windows-style line endings
  • Updated both CsvWriter and ArrowToCsvWriter to support line ending configuration
  • Added comprehensive test coverage for both line ending types
  • Updated documentation to reflect the new configuration option

Technical Details

  • Added CsvLineEnding type to configuration specs
  • Implemented custom CSV dialect for Python's csv writer to handle CRLF
  • Added version-aware implementation for PyArrow CSV writer (supports line endings from version 14.0)
  • Added configuration options in both TOML and environment variables

Testing

  • Added unit tests for both CsvWriter and ArrowToCsvWriter
  • Verified correct line ending generation for both LF and CRLF
  • Ensured no mixed line endings in output files
  • Added tests for both direct data writing and Arrow table conversion

Documentation

  • Updated CSV format documentation to include the new line_ending option
  • Added examples for both TOML and environment variable configurations

Configuration Example

[NORMALIZE]
data_writer__line_ending="crlf"

Or via environment variable:

NORMALIZE__DATA_WRITER__LINE_ENDING=crlf

Checklist

  • Added new configuration option
  • Implemented support in both CSV writers
  • Added comprehensive tests
  • Updated documentation
  • Verified backward compatibility
  • Added examples for configuration

Copy link

netlify bot commented Jun 18, 2025

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit 2f817b5
🔍 Latest deploy log https://app.netlify.com/projects/dlt-hub-docs/deploys/6852d4e9ed3fc00008d60621
😎 Deploy Preview https://deploy-preview-2783--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant