rejig

Unicode Usage
Login

Unicode Usage

One of the driving motivations behind the creation of rejig was my dismay over the nonstandard R47 character set, coupled with a presumption that no one will volunteer to undertake the huge task of resolving all the discrepancies. After much discussion on the forum, that appears to be an accurate view.

And that’s perfectly fine! Unlike the R47 project proper, rejig doesn’t have a choice in this matter: its default output format is UTF-8 plaintext, where we cannot count on any particular font being used. It must of necessity conform to the Unicode standard.

The solution is to accept R47 programs in byte code form, then translate them into proper Unicode.

Character Sets

The idealized form of all text strings within rejig is pure UTF-8. Just as the R47 instruction set provides our lingua franca for ops and their arg types, Unicode provides our lingua franca for alpha string code points. Everything below raises to this form on input, then lowers from this form on output.

What all this means in consequence, within the rejig sphere, is that this is an illegal program line:

42α ‘«quote?»’

Why? The leading guillemet is encoded as the nonstandard Free42 character 0x88, and because it has the top bit set, it is misinterpreted as a VIEW IND op! Wishing to keep end users from falling into this trap, rejig screams and dies when you try this.

While I could choose to allow this within rejig as a form of synthetic programming, I thought that was a crock on the HP-41C, an opinion HP appears to agree with given how much effort they went to in shutting that pathway down in the HP-42S. I refuse to reopen that door to chaos within rejig for non-HP-41C targets. Indeed, it’s one reason I hesitated for so long to add HP-41C support at all.

A related coding ambiguity happens when leading an alpha string with the ⊢ character. Since this is how the HP-42S signals an “alpha append” operation, rejig diagnoses this condition as a fatal error for the same basic reason as above. It remains perfectly legal to put that non-ASCII character in the middle of an alpha string, or at the end. Getting it to appear at the start of an alpha string is possible, but it suffices to say that rejig supports the hacky construction required.

It gets weirder: HP gave that append-alpha op byte code 0x7F, which ASCII instead assigned for good and proper reasons to the DEL (rubout) character. HP then went and added a rubout-looking character (▒) at 0x04, overriding the spot ASCII assigned to the EoT, entered on any ASCII terminal as Ctrl-D. This is why that particular keystroke exits any sensible CLI app, from command shells onward. The thing is, sending that character to the 82240A will neither end the printing session nor display a rubout character in a teletype fashion, overwriting a prior printed character. No, it advances the paper one line while leaving the printhead where it is! This is the behavioral definition of line feed, ASCII LF, 0x10, yet HP in its wisdom defined 0x10 as having the effect of CR-LF on the 82240A, both advancing the paper one line and sending the printhead back to the start of the line.

(It should not surprise me to learn that HP’s engineering departments were packed with autists who never spoke to each other even when working on companion products whose manuals reference one another, but here we have the proof.)

I will close out this section with what little reassurance I can offer, given the circumstances: rejig does its best to hide this four-way fight from those of you innocent bystanders just trying to get useful work done. What I cannot promise is that you will never someday find yourself beyond the margins, gawping with dismay at all this mess. Sorry; how it is.

Practical Transformations

The best user experience for typing Unicode is on macOS, and even there, getting exactly the right character isn’t always easy. We therefore provide an assortment of transformations within rejig to ease manual program entry:

These are used not only in expressing P47 byte codes in rejig but may also be used in end user programs, honored in parsing. Because rejig converts all input to a canonical form, you can use it to “upgrade” a plain ASCII version of a program using the in-place reformatting feature.

File Encoding

Text inputs must be Unicode, in any of these encodings:

We do not make any attempt to handle UTF-7, nor legacy 8-bit ASCII extensions like CP1252 or ISO-8859.

The --from/to options utf8 and unicode are synonyms. The sole consequence of specifying --from utf8 while giving non-UTF-8 Unicode input is that rejig issues a warning; decoding continues.

Windows CR+LF line endings are accepted on input, but text output prefers the Unix LF style.

(You may now wish to return to the rejig home page.)

License

This work is © 2026 by Warren Young and is licensed under CC BY-NC-SA 4.0