unicode_data

VivoMind Prolog Unicode Resources

Authors

Arun Majumdar @ VivoMind LLC
Paulo Moura @ VivoMind LLC

License

Creative Commons CC0 1.0 Universal (CC0 1.0) - Public Domain Dedication:

https://creativecommons.org/publicdomain/zero/1.0/

We do appreciate acknowledgment if you use these resources, however, and we also welcome contributions to improve them.

Website

The latest release of the VivoMind Prolog Unicode Resources is available at the URL:

https://github.com/VivoMind

At this address you can also find additional information about the VivoMind Prolog Unicode Resources and submit your bug reports and contributions.

Description

This library contains generated Unicode 17.0.0 resources. The unicode_character_data.pl file provides normalization, default casing, and core text-classification predicates in user for use by the text_normalization library. The legacy Prolog APIs are generated from the same pinned Unicode release.

Frequently shared range families use generated backing predicates and thin compatibility views. This avoids storing the same Unicode ranges in each legacy property file while preserving the existing filenames, predicate indicators, enumeration behavior, and default values. Independent property families remain selectively loadable. The original data can be downloaded from:

http://www.unicode.org

The conversion of the UCD files resulted in a large number of Prolog tables and also a set of auxiliary predicates (described below) for accessing these tables. Other than the obvious conversion in the provided predicate names, no attempt was made to convert the identifiers used for properties and other data.

Requirements

Most of the auxiliary predicates assume that the de facto Prolog standard predicate between/3 is available. Unicode code point values are represented as integers. In addition, the ISO Prolog standard directives include/1 and ensure_loaded/1 are used in some files to load auxiliary files.

Usage

To load the optimized Unicode 17.0.0 data used for text normalization and default case conversion:

| ?- logtalk_load(unicode_data(loader)).

Logtalk entities can call these predicates directly using a uses/2 directive for user.

To load all legacy compatibility APIs:

| ?- logtalk_load(unicode_data(unicode_data)).

Most applications only require some of the tables present in these resources. Most of these tables define properties for ranges of code points and not for single code points, but the provided auxiliary predicates allow access for a single code point. When increased performance is required, consider using the existing tables and auxiliary predicates to generate derived tables more fit for your specific application.

Known issues

The include/1 and ensure_loaded/1 directives are specified in the ISO Prolog standard published in 1995. But some Prolog compilers either don’t implement one or both directives or have flawed implementations. Thus, you may need to change how some of the files are loaded depending on the chosen Prolog compiler. Using conditional compilation directives would help in some cases, but it would also raise portability issues on its own.

Acknowledgements

We thank Richard O’Keefe for helpful suggestions to improve the usability of these resources.

Files and API Summary

The Prolog file names are derived from the original file names by prefixing them with the unicode_ string, converting to lower case, and replacing the camel case spelling with underscores. There are, however, two exceptions: the files and directories holding the code point categories and names.

There’s also a utility file, unicode_data.pl, that can be used to load all the files in these resources. It is mostly used to test the portability of the code across Prolog compilers. Also included is a Logtalk version of this file, unicode_data.lgt, which uses Logtalk’s own implementation of the include/1 directive and the logtalk_load/1 predicate to load all files. This file can be used to workaround Prolog systems with buggy or missing implementations of the ensure_loaded/1 and include/1 directives.

An overview of the original file names and the code point properties can be found at:

http://www.unicode.org/reports/tr44/#Directory_Structure http://www.unicode.org/reports/tr44/#Property_Definitions

unicode_arabic_shaping.pl

  • Provides: unicode_arabic_shaping/4

  • Dependencies: (none)

unicode_bidi_mirroring.pl

  • Provides: unicode_bidi_mirroring/2

  • Dependencies: (none)

unicode_blocks.pl

  • Provides: unicode_block/2-3

  • Dependencies: (none)

unicode_case_folding.pl

  • Provides: unicode_case_folding/3

  • Dependencies: (none)

unicode_categories.pl

  • Provides: unicode_category/2

  • Dependencies: files in the unicode_categories directory

unicode_cjk_radicals.pl

  • Provides: unicode_cjk_radical/3

  • Dependencies: (none)

unicode_composition_exclusions.pl

  • Provides: unicode_composition_exclusion/1

  • Dependencies: (none)

unicode_core_properties.pl

  • Provides: unicode_math/1-2 unicode_alphabetic/1-2 unicode_range_alphabetic/2 unicode_lowercase/1-2 unicode_uppercase/1-2 unicode_cased/1-2 unicode_case_ignorable/1-2 unicode_changes_when_lowercased/1-2 unicode_changes_when_uppercased/1-2 unicode_changes_when_titlecased/1-2 unicode_changes_when_casefolded/1-2 unicode_changes_when_casemapped/1-2 unicode_id_start/1-2 unicode_id_continue/1-2 unicode_xid_start/1-2 unicode_xid_continue/1-2 unicode_default_ignorable/1-2 unicode_grapheme_extend/1-2 unicode_grapheme_base/1-2 unicode_grapheme_link/1-2

  • Dependencies: files in the unicode_core_properties directory

unicode_decomposition_type.pl

  • Provides: unicode_canonical/1-2 unicode_compat/1-2 unicode_font/1-2 unicode_nobreak/1-2 unicode_initial/1-2 unicode_medial/1-2 unicode_final/1-2 unicode_isolated/1-2 unicode_circle/1-2 unicode_super/1-2 unicode_sub/1-2 unicode_vertical/1-2 unicode_wide/1-2 unicode_narrow/1-2 unicode_small/1-2 unicode_square/1-2 unicode_fraction/1-2

  • Dependencies: files in the unicode_decomposition_type directory

unicode_derived_age.pl

  • Provides: unicode_age/2-3

  • Dependencies: (none)

unicode_derived_bidi_class.pl

  • Provides: unicode_bidi_class/2-3

  • Dependencies: (none)

unicode_derived_combining_class.pl

  • Provides: unicode_combining_class/2-3

  • Dependencies: (none)

unicode_derived_core_properties.pl

  • Provides: unicode_core_property/2-3

  • Dependencies: (none)

unicode_derived_decomposition_type.pl

  • Provides: unicode_decomposition_type/2-3

  • Dependencies: (none)

unicode_derived_east_asian_width.pl

  • Provides: unicode_east_asian_width/2-3

  • Dependencies: (none)

unicode_derived_joining_group.pl

  • Provides: unicode_joining_group/2-3

  • Dependencies: (none)

unicode_derived_joining_type.pl

  • Provides: unicode_joining_type/2-3

  • Dependencies: (none)

unicode_derived_line_break.pl

  • Provides: unicode_line_break/2-3

  • Dependencies: (none)

unicode_derived_normalization_props.pl

  • Provides: unicode_fc_nfkc/2 unicode_nfkc_cf/2 unicode_full_composition_exclusion/1-2 unicode_nfd_qc_no/1-2 unicode_nfc_qc_no/1-2 unicode_nfc_qc_maybe/1-2 unicode_nfkd_qc_no/1-2 unicode_nfkc_qc_no/1-2 unicode_nfkc_qc_maybe/1-2 unicode_expands_on_nfd/1-2 unicode_expands_on_nfc/1-2 unicode_expands_on_nfkd/1-2 unicode_expands_on_nfkc/1-2 unicode_changes_when_nfkc_casefolded/1-2

  • Dependencies: files in the unicode_derived_normalization_props directory

unicode_derived_numeric_type.pl

  • Provides: unicode_numeric_type/2-3

  • Dependencies: (none)

unicode_derived_numeric_values.pl

  • Provides: unicode_numerical_value/3

  • Dependencies: (none)

unicode_hangul_syllable_type.pl

  • Provides: unicode_hangul_syllable_type/2-3

  • Dependencies: (none)

unicode_indic_matra_category.pl

  • Provides: unicode_indic_matra_category/2-3

  • Dependencies: (none)

unicode_indic_syllabic_category.pl

  • Provides: unicode_indic_syllabic_category/2-3

  • Dependencies: (none)

unicode_jamo.pl

  • Provides: unicode_jamo/2

  • Dependencies: (none)

unicode_name_aliases.pl

  • Provides: unicode_name_alias/3

  • Dependencies: (none)

unicode_names.pl

  • Provides: unicode_name/2

  • Dependencies: files in the unicode_names directory

unicode_prop_list.pl

  • Provides: unicode_white_space/1-2 unicode_bidi_control/1-2 unicode_join_control/1-2 unicode_dash/1-2 unicode_hyphen/1-2 unicode_quotation_mark/1-2 unicode_terminal_punctuation/1-2 unicode_other_math/1-2 unicode_hex_digit/1-2 unicode_ascii_hex_digit/1-2 unicode_other_alphabetic/1-2 unicode_ideographic/1-2 unicode_diacritic/1-2 unicode_extender/1-2 unicode_other_lowercase/1-2 unicode_other_uppercase/1-2 unicode_noncharacter_code_point/1-2 unicode_other_grapheme_extend/1-2 unicode_ids_binary_operator/1-2 unicode_ids_trinary_operator/1-2 unicode_radical/1-2 unicode_unified_ideograph/1-2 unicode_other_default_ignorable/1-2 unicode_deprecated/1-2 unicode_soft_dotted/1-2 unicode_logical_order_exception/1-2 unicode_other_id_start/1-2 unicode_other_id_continue/1-2 unicode_sterm/1-2 unicode_variation_selector/1-2 unicode_pattern_white_space/1-2 unicode_pattern_syntax/1-2

  • Dependencies: files in the unicode_prop_list directory

unicode_range_scripts.pl

  • Provides: unicode_range_script/3 unicode_script/2

  • Dependencies: (none)

unicode_script_extensions.pl

  • Provides: unicode_script_extension/2-3

  • Dependencies: unicode_scripts.pl

unicode_scripts.pl

  • Provides: unicode_script/6 unicode_script_category/3

  • Dependencies: (none)

unicode_special_casing.pl

  • Provides: unicode_special_casing/5

  • Dependencies: (none)

unicode_unihan_variants.pl

  • Provides: unicode_unihan_variant/2-3

  • Dependencies: (none)

unicode_version.pl

  • Provides: unicode_version/3

  • Dependencies: (none)