I've reviewed the private thread inventorying work items before a merge of os8-v3f-extensions into trunk. (That thread ran from 23 July 2-18 to 13 August 2018.)

Questions Answered:

Q:	Might this pending event order reversal change semantics affecting the current scripts?
A:	Only in an upwardly compatible way: Currently all actions are performed without the ability to confirm success during the script. Either the actions succeed, or a failure results in ignored action or dead script. When the event order reversal happens, then we get to add rich tests for the success/failure of actions.

Q:	What are the different running packs to be offered and why?
A:	It could be a choice among v3d/v3f patches/no-patches. Because v3d is not far enough along, it is offered as a bootable tu56. Because best practice is keep a read-only pristine boot pack without patches, but to run with all mandatory patches in place we offer:
v3d-dist.rk05	Read-only un-patched v3d distribution media.
v3d.rk05	Patched version of v3d-dist.rk05 -- the system you always use.
v3f-tc08.tu56	Bootable DECtape image. One of 4 for the pairings v3d/v3f tco8/td12k

Q:	Should we not rename os8.tu56?
A:	Yes. Actually os8.tu56 is a mess and should simply go away.

Q:	What is the file name for the final V3F output media.
A:	Two files: v3f-td12k.tu56, v3f-tc08.tu56

Q:	Are the media inputs for v3f still changing?
A:	I can freeze work on v3f, "across testing in anticipation of the merge." As said elsewhere, v3f is expected to continue to evolve after the merge, because the merge is for a public release of a v3f framework, not a completed v3f deliverable.

Q:	Should we abandon the convention that IF=3 is a DECtape boot?
A:	No. But now we have a framework for configuring what gets booted. One of 4 .tu56 images: v3d-tc08.tu56, v3d-td12k.tu56, v3f-tc08.tu56, v3f-td12k.tu56.

Actions:

• DONE: os8-run should have an independent version number, with the major number indicating stable syntax versions (Version 1.0 established and version-based conditionals implemented.)

• DONE: Simplification of the version-based conditionalization.

• FIXED: Note that rebuid of os8v3d-src.rk05 seems to happen more than I expect. (There was a dependency mistake that I finally found and fixed.)

• FIXED: "make ; make" results in two separate os8-run calls.

• DONE: Optional: New command to exit a script, sending exit status integer to the calling POSIX shell.

• DONE: Framework to decide on whether v3d or v3f gets booted by default. Proof of concept for DECtape images.

• DONE: Shorter, more obvious names for all output image files. (Of particular value is calling the unpatched v3d binary "dist" for "Distribution" signifying it's a read-only system, and renaming "patched" to be simply "v3d" meaning "The one you should always use." Wanting the patched v3d for building v3f, plus recognition that conventional best practices were to keep a read-only master image, and then to have the running system contain all mandatory patches.)

• DONE: Cleanup of underscores in filenames.

• DONE: Rename os8run scripts to the .os8 extension instead of .mkos8.

• DONE: Create framework for choice between TD8E and TC08 for creation of .tu56 images, for choice of running them, and for robust building of them. (Lots of learning. Lots of evolution. We build 4 .tu56 images. We use a scratch boot rk05 when we're going to build for TD8E. The TD8E drivers are now part of the default rk05 BUILD.SV image. )

• DONE: New os8-run commands for printing status messages. It understands and can print the syntax version number and the pathname prefixes that are in force at run time.

• DONE: Init scripts for playing with v3f: Conditionals to set a v3f tu56 image on IF=3.

• FIXED: "make mediainstall" does not yet contain a command to copy bin/*.tu56.

• FIXED: The present TCO8 build of v3f does not have a very full complement of executable files.

Out of scope:

• Rewrite of event loop.

• rk05 image with complete v3f system -- I don't want to gate this merge on the "fully qualified and tested v3f. Instead this is the v3f playpen version.

• Adoption of Tcl as the scripting environment. After I learned more about how autodef worked and its Tcl roots, I learned more of Tcl. I don't like it. I won't say I hate it like I hate perl, but it offends some of my sensibilities in the same way perl does. For example variables beginning with a dollar sign. And certain bits of state that are implicit in the lore of programming in the language, rather than explicitly declared in code. Oh yes and like perl, defined interfaces seemingly to not work at all. (opt-val string value defaults.) You said, "Tcl's rules for this are complex and powerful, but it allows a style that looks like csh," like it was a good thing. I had trouble with the substitution stuff. It seemed too clever and idiosyncratic for my tastes.

• OS/8 packaging system. This branch has become "the thing that lets you use SIMH to perform elaborate scripting within OS/8 to build arbitrary bootable media." To get back to, "Make me packs with FOCAL but not ADVENT." we need to come at the problem afresh, I think. But the config options currently in place still do a pretty good job of package management. Funny thing: With the begin/end enabled <arbitrary string> we CAN create scripts that take a string of arguments as what to put in. The dependencies would end up being explicit in the scripts.

Open Issues:

1. Is the creation of the v3f-build.rk05 and v3f-made.rk05 still in need of further cleanup/refinement? The opinion was expressed, "the -build and -made files should be written to the obj subdir, not bin. Everything in bin ends up installed, but I don't think we want to install -made or -build." I actually think v3f-made.rk05 should be installed for users to play with. I'm coming to believe v3f-build.rk05 belongs in obj. Mind you, touching a single source file in .../src/os8/v3f/ causes BOTH to be rebuilt from scratch.

2. Do we want a "run-v3f" target for Makefile.in?

3. What kind of validation should we run to assure ourselves that os8-run really has successfully replaced mkos8 for the os8v3d system disk images? It was suggested, "decomposing each disk image into a set of files using os8-cp would work. Then instead of comparing whole medium images, we'd compare each file to the corresponding file in the test corpus. " That would definitely be doable. The os8-cp action file is totally your friend here!

Remaining:

1. Remove os8.tu56, and the dependency on it for make mediainstall.

2. Rewrite of the automated tester for two outcomes:

   1. Elimination of all mkos8 vestiges.
   2. Re-implementation of successful test suite.

3. Run-through of test-os8-run with few enough problems that we feel comfortable.

4. Cleanup / deletion of any remaining build code that thinks patching is optional.

5. Creation of init.tx files for the 4 bootable tu56 images.

6. Confirm that IF=3 actually DOES boot all 4 bootable tu56 images.

Notes on test-mkos8:

All test-mkos8 does is:

1. On the test corpus creation pass, ensure that no combination of configure script options results in an incomplete build. It relies on the Makefile rules to ensure that if the build breaks, we get no final output file. This means build errors can't be ignored, resulting in degenerate cases like 0-byte final output media. A build error must stop processing immediately, not stumble forward blindly, creating errors upon errors.

2. When running verification tests against that corpus, it checks that builds generate binary-identical copies of the media given the same configure script options.

Think of test-mkos8 as an integration test system, not as a unit test system. It doesn't care about things like how OS/8 V3F PIP work given certain commands, it only cares that the OS/8 binary media is identical from one run to the next, implying that programs like PIP are binary-identical and in the same place on the output medium image. That is test-mkos8's definition of "correct."

There was a dependency mistake that I finally found and fixed.

Parallel builds (e.g. make -j6) can now fail with this error:

At line 54, /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3d.rk05 must exist but was not found. Not mounting.

I haven't bothered to scan the timeline to find your fix and see if it's related, but it's plausible that it might be. If so, then there's probably a different fix that solves both problems.

I don't like [Tcl]

That's fine, but that puts the change more than just "out of scope," but probably more like "out of all reasonable possibility," since rewriting os8-run atop Tcl would probably change its syntax and would certainly change its semantics. Therefore, unless someone else pops up and does that rewrite before release, it would constitute a breaking change that even calling it "v2" might not cover. Since you're not willing and I have only a little interest in the idea, I think it's not going to happen.

I won't argue with you about the merits and demerits of Tcl here, since neither of us is going to do the rewrite, so it isn't on topic here. We can discuss it via private email, if you like, but my only interest would be in correcting misapprehensions, rather than try to make you a Tcl convert. I offered the idea only as a way to avoid a whole lot of parsing and wheel reinvention. As I recall, the idea came up when we were talking about adding conditionals to the os8-run language, which you'd have gotten for free with the redesign.

Do we want a "run-v3f" target for Makefile.in?

Yes, so I added it. See also my changes to 3.script.in.

Remove os8.tu56, and the dependency on it for make mediainstall.

Done.

The present TCO8 build of v3f does not have a very full complement of executable files.

This might need more work. Here's a unified diff of the sorted DIR output from v3f-tc08.tu56 to that from os8.tu56:

ABSLDR.SV
 BATCH .SV
+BCOMP .SV
 BITMAP.SV
+BLOAD .SV
 BOOT  .SV
-BUILD .SV
+BRTS  .SV
 CCL   .SV
 CREF  .SV
 DIRECT.SV
-DTCOPY.SV
-DTFRMT.SV
+ECHO  .SV
 EDIT  .SV
 EPIC  .SV
+F4    .SV
+FBOOT .SV
+FORLIB.RL
 FOTP  .SV
+FRTS  .SV
 FUTIL .SV
-HELP  .HL
-HELP  .SV
+IDS   .SV
+LCSYS .BI
+LIBRA .SV
+LOAD  .SV
 PAL8  .SV
+PASS2 .SV
+PASS2O.SV
+PASS3 .SV
 PIP   .SV
+PT8E  .BN
+RALF  .SV
 RESORC.SV
-RXCOPY.SV
+SABR  .SV
 SET   .SV
-SRCCOM.SV
+TD8EDT.BN
+TDCOPY.SV
+TDFRMT.SV
 TECO  .SV
+TEST  .FT
+TEST  .LD
+TEST  .RL
+UCSYS .BI

It looks like the major differences are BASIC, FORTRAN II, and FORTRAN IV. I think the presence of these should be controlled by the same options that control whether the OS/8 RK05 disk gets these features.

The --lowercase flag should also take effect in the tape image.

The removal of TEST.* is probably a good thing.

I'm getting os8-run errors from a single-threaded make on Raspbian 9:

Running script file: ./media/os8/scripts/all-tu56.os8
Building v3f
with TC08 support.
Pre-existing /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3f-tc08.tu56 found.  Saving as /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3f-tc08.tu56.save
Traceback (most recent call last):
  File "bin/os8-run", line 222, in <module>
    main()
  File "bin/os8-run", line 206, in main
    os8.run_script_file (script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 915, in run_script_file
    retval = commands[m.group(1)](rest, script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1400, in begin_command
    return sub_commands[m.group(1)](m.group(3), script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1542, in build_subcomm
    reply = self.simh._child.expect("SYS BUILT")
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x769ec270>
command: /usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim
args: ['/usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim']
buffer (last 100 chars): 'OOT \r\nOOT?\r\n$'
before (last 100 chars): 'OOT \r\nOOT?\r\n$'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 30483
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <open file '/usr/local/src/pidp8i/os8-v3f-extensions/obj/os8-run.log', mode 'w' at 0x7652d230>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: None
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile("SYS BUILT")
Makefile:564: recipe for target 'bin/v3f-tc08.tu56' failed
make[1]: *** [bin/v3f-tc08.tu56] Error 1
make[1]: Leaving directory '/usr/local/src/pidp8i/os8-v3f-extensions'
bin/os8-run ./media/os8/scripts/all-tu56.os8 --enable v3f --enable td12k
Running script file: ./media/os8/scripts/all-tu56.os8
Building v3f
for td12k configuration of TD8E.
Pre-existing /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3f-td12k.tu56 found.  Saving as /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3f-td12k.tu56.save
Traceback (most recent call last):
  File "bin/os8-run", line 222, in <module>
    main()
  File "bin/os8-run", line 206, in main
    os8.run_script_file (script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 915, in run_script_file
    retval = commands[m.group(1)](rest, script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1400, in begin_command
    return sub_commands[m.group(1)](m.group(3), script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1542, in build_subcomm
    reply = self.simh._child.expect("SYS BUILT")
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x76a4f2b0>
command: /usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim
args: ['/usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim']
buffer (last 100 chars): 'OT \r\nOT?\r\n$'
before (last 100 chars): 'OT \r\nOT?\r\n$'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 30491
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <open file '/usr/local/src/pidp8i/os8-v3f-extensions/obj/os8-run.log', mode 'w' at 0x76590230>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: None
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile("SYS BUILT")
Makefile:568: recipe for target 'bin/v3f-td12k.tu56' failed
make: *** [bin/v3f-td12k.tu56] Error 1

I've tried reconfiguring, removing configure script options, and re-making from a clean state.

Re: Tcl: We're both on the same page here.

It was definitely worth thinking about.

Perhaps my initial negative impression of will fade. I've too often gotten caught up in strong opinions that proved obstacles to simple forward progress.

But I think "not going to happen" as the bottom line of "convert to Tcl" is pragmatic, and correct.

Re: remaking the source rk05:

My fix is in artifact 48bf62dd63 . But root cause was: New v3d-src.rk05 kept getting created with a version number suffix , and the old file was NEVER replaced. The test that would see an updated v3d-src.rk05 image had been created as a consequence of the dependencies would ALWAYS fail.

The dependency problem you are having has nothing to do with the v3d-src.rk05 image. It's complaining about needing v3d.rk05, the "image you always run."

That image is a prerequisite for any v3f work. So you can't do any v3f work until:

1. v3d-dist.rk05 containing the read-only baseline run image is created.
2. A copy of v3d-dist.rk05 is created that is then patched to create v3d.rk05, the "image you always run."

I suspect that in Makefile.in dependencies were established for the image os8v3d-bin.rk05 and one not two changes got made.

1. (DONE) The os8v3d-bin.rk05 got renamed to v3d-dist.rk05 with the new naming convention.
2. Names in dependencies should have changed from v3d-dist.rk05 to v3d.rk05 when I adopted the new convention that we DON'T use the unpatched images for subsequent work.

I will investigate Makefile.in for such a mistake.

Re: run-v3f:

ok.

Re: Contents of the new IF=3 tu56 image.

I did what I said I was going to do: Make a minimal boot tape, presuming Fortran II, Fortran IV, and BASIC were not going to be present.

In retrospect, yes there should be configuration options to include one or more of them on the tape.

I think we want those separated from the present package selection options because there's a LOT less space in the tu56 image, and if you have all 3 there's basically no room to do anything.

Re: --lowercase:

It's easy to add it to the v3d image. Because it's a binary patch into code the MOVED in v3f, a DIFFERENT patch is required for v3f.

Think about if you want --lowercase to affect ALL builds, and if so I can do the needful.

Re: Re: make run-v3f:

Do you realize that you've got the dependency solely on the one run image v3f-tc08.tu56, but you're using the boot script that is conditionalized to be able to use either that image or the v3f-td12k.tu56 image?

You probably want it to be dependent on both images.

Would you send me the output of:

bin/os8-run -d ./media/os8/scripts/all-tu56.os8 --enable v3f --enable td12k

The -d option is added which will print me out some helpful stuff.

There's some icky code in the procedure build_subcom that's many stupid special cases to work around how the main event loop is backwards.

Every time I turn around there's some damn thing with the expect routines where it's not getting something, and the fault lies in that particular block of code, near line 1532 needing yet one more kludge.

We will compare the Raspbian 9's debug output with what we get on the other platforms and see what platform-specific special character is messed up in the regular expressions we're sending to pexpect.

Re: failed parallel build dependency:

I just checked in a change to Makefile.in where we were depending on the v3d-dist.rk05 image when we should have been depending on the run image v3d.rk05.

See if that fixes the parallel make.

Here's the top part:

options_enabled['v3f', 'td12k']
options_disabled[]
Running script file: ./media/os8/scripts/all-tu56.os8
options_enabled: ['v3f', 'td12k', 'v3d']
options_disabled: []
options_stack: []
Pushing enabled block v3f onto options_stack
new options_stack: ['v3f']
Popping v3f
new options_stack: []
options_enabled: ['v3f', 'td12k']
options_disabled: ['v3d']
options_stack: []
Pushing enabled block v3f onto options_stack
new options_stack: ['v3f']
Building v3f
Popping v3f
new options_stack: []
options_enabled: ['v3f', 'td12k']
options_disabled: ['v3d']
options_stack: []
ignore to: enabled v3d
options_enabled: ['v3f', 'td12k']
options_disabled: ['v3d']
options_stack: []
Pushing enabled block v3f onto options_stack
new options_stack: ['v3f']
Popping v3f
new options_stack: []
options_enabled: ['v3f', 'td12k', 'tc08']
options_disabled: ['v3d']
options_stack: []
Pushing enabled block td12k onto options_stack
new options_stack: ['td12k']
for td12k configuration of TD8E.
Popping td12k
new options_stack: []
options_enabled: ['v3f', 'td12k', 'tc08', 'td8e']
options_disabled: ['v3d']
options_stack: []
ignore to: enabled tdrom
options_enabled: ['v3f', 'td12k', 'tc08', 'td8e']
options_disabled: ['v3d']
options_stack: []
Pushing enabled block td8e onto options_stack
new options_stack: ['td8e']
Popping td8e
new options_stack: []
options_enabled: ['v3f', 'td12k', 'td8e']
options_disabled: ['v3d', 'tc08']
options_stack: []
ignore to: enabled tc08
options_enabled: ['v3f', 'td12k', 'td8e']
options_disabled: ['v3d', 'tc08']
options_stack: []
Pushing enabled block td8e onto options_stack
new options_stack: ['td8e']

I'm skipping a bunch of noisy-looking stuff in the middle.

This much of the end looks like it might be helpful:

sending to simh: BOOT 
expecting: ['\\$', 'SYS BUILT', 'WRITE ZERO DIRECT\\?', '\\?BAD ARG', '\\?BAD INPUT', '\\?BAD LOAD', '\\?BAD ORIGIN', '\\?CORE', '\\?DSK', '\\?HANDLERS', 'I/O ERR', '\\?NAME', 'NO ROOM', 'SYS NOT FOUND', '\\?PLAT', '\\?SYNTAX', '\\?SYS', 'SYS ERR', '\\S+ NOT FOUND']
reply: 0
before: DSK:CD.BN
after: $
Traceback (most recent call last):
  File "bin/os8-run", line 222, in <module>
    main()
  File "bin/os8-run", line 206, in main
    os8.run_script_file (script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 915, in run_script_file
    retval = commands[m.group(1)](rest, script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1400, in begin_command
    return sub_commands[m.group(1)](m.group(3), script_file)
  File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/os8script.py", line 1542, in build_subcomm
    reply = self.simh._child.expect("SYS BUILT")
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/local/lib/python2.7/dist-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: Timeout exceeded.
<pexpect.pty_spawn.spawn object at 0x76a47230>
command: /usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim
args: ['/usr/local/src/pidp8i/os8-v3f-extensions/bin/pidp8i-sim']
buffer (last 100 chars): 'OT \r\nOT?\r\n$'
before (last 100 chars): 'OT \r\nOT?\r\n$'
after: <class 'pexpect.exceptions.TIMEOUT'>
match: None
match_index: None
exitstatus: None
flag_eof: False
pid: 6286
child_fd: 5
closed: False
timeout: 30
delimiter: <class 'pexpect.exceptions.EOF'>
logfile: <open file '/usr/local/src/pidp8i/os8-v3f-extensions/obj/os8-run.log', mode 'w' at 0x765c3230>
logfile_read: None
logfile_send: None
maxread: 2000
ignorecase: False
searchwindowsize: None
delaybeforesend: None
delayafterclose: 0.1
delayafterterminate: 0.1
searcher: searcher_re:
    0: re.compile("SYS BUILT")

Since posting the above, I tried it on a Debian 9 x86 VM — which should be pretty close to Raspbian on a Pi, differing largely in CPU type — and it built without error.

The Pi appears to have pexpect 4.2.1 on it, if that helps.

I've got a different symptom now, which appears unrelated:

cd ./src/os8/v3f; /usr/local/src/pidp8i/os8-v3f-extensions/bin/os8-cp -v --action-file actions.txt
Abort: Requested boot image file: /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3d.rk05 not found.
make: *** [bin/v3f-build.rk05] Error 255

No, I didn't realize that. Good catch; it's fixed now.

It's not depending on both images, because you don't want to force both to be created if both aren't needed. It just uses the same autosetup variable as goes into boot/3.script.

Actually, that fix creates another problem: the OS8_BOOT_TAPE variable picks between FOUR tapes:

• v3d-tc08.tu56
• v3d-td12k.tu56
• v3f-tc08.tu56
• v3f-tc12k.tu56

I'd set the dependencies for make all to create all four.

I defined two config parameters:

boot-tape-config:  => "Boot tape configuration: tc08, or td12k"
boot-tape-version: => "OS/8 version for boot tape, either v3d or v3f"

I conceived of OS8_BOOT_TAPE as "Which of the 4 images you get when you set IF=3."

To get what I think you want with make run-v3f we need to create a different variable to operate on in Makefile.in that uses boot-tape-config, but NOT boot-tape-version:

V3F_BOOT_TAPE= bin/v3f-@OS8_TAPE_DEVICE@.tu56

I have checked that amendment into your amendment.

I think I found and fixed this one. The build of v3f-build.rk05 lacked a dependency on OS8_BOOT_DISK.

I just checked in this:

# Build the source disk for OS/8 V3F
bin/$(V3F_BUILD_RK05): $(V3F_SRCDIR)/$(V3F_MANIFEST) $(V3F_SOURCES) $(OS8_BOOT_DISK)
	cd $(V3F_SRCDIR); @builddir@/$(OS8CP) -v --action-file $(V3F_MANIFEST)
	mv $(V3F_SRCDIR)/$(V3F_BUILD_RK05) bin

I've done some more experimentation with what will fit on a single tu56 image.

BASIC and FORTRAN II will fit, but FORTRAN IV has a big library that hoovers up all the space.

The newly checked in defaults put BASIC and FORTRAN II on the bootable tapes.

I included one demo program HELLO.BA, and tested it on the v3f-tc08.tu56 image.

TODO:

1. Test out FORTRAN II and make sure it works.
2. Test all 4 tu56 images.
3. Consider including a test FORTRAN II Program.
4. Consider adding a tape option to replace FORTRAN II and BASIC with FORTRAN IV.

I was ready to accept that the TU56 tapes just wouldn't have these things.

We could just document that they're stripped down to basics on purpose, then do the full-boat feature build in the RK05 media for V3F.

It looks like there's more left to do on this. With the current tip of the branch, I get another complaint about that file now:

Abort: Requested boot image file: /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3d.rk05 not found.
Running script file: ./media/os8/scripts/v3d-dist-rk05.os8
make: *** [bin/v3f-build.rk05] Error 255
make: *** Waiting for unfinished jobs....

This happens shortly after it finishes linking the simulator and creating the dynamic boot/*.script files from the example programs and such.

Reproduction steps: make clean && tools/mmake on macOS.

I haven't tried it on Raspbian again yet.

I've checked in an update that adds init.cm and appropriate init.tx for v3f to the tc08 and td12k tu56 tape images for v3f. It also puts the init.cm and init.tx to the tc08 and td12k tu56 image for v3d.

This implements item 5 from the section "Remaining."

What's our gap now?

I think we have not yet come to closure on two of the 3 open issues:

1. Is the creation of the v3f-build.rk05 and v3f-made.rk05 still in need of further cleanup/refinement? The opinion was expressed, "the -build and -made files should be written to the obj subdir, not bin. Everything in bin ends up installed, but I don't think we want to install -made or -build." I actually think v3f-made.rk05 should be installed for users to play with. I'm coming to believe v3f-build.rk05 belongs in obj. Mind you, touching a single source file in .../src/os8/v3f/ causes BOTH to be rebuilt from scratch.

Q1: With regards to the v3f-build/v3f-made disks, there's one funky dependency:

We depend on OS8_BOOT_DISK, but RUNNING that disk touches it. I see that there's a kind of dependency, an "order only prerequisite" that I've heretofore never used. Should I change Makefile.in

bin/$(V3F_BUILD_RK05): $(V3F_SRCDIR)/$(V3F_MANIFEST) $(V3F_SOURCES) $(OS8_BOOT_DISK)

to put a vertical bar before "$OS8_BOOT_DISK"?

Q2: Are we agreed that v3f-build.rk05 should move to obj, but that v3f-made.rk05 should stay in bin for installation as the "v3f source and build running playpen"?

3. What kind of validation should we run to assure ourselves that os8-run really has successfully replaced mkos8 for the os8v3d system disk images? It was suggested, "decomposing each disk image into a set of files using os8-cp would work. Then instead of comparing whole medium images, we'd compare each file to the corresponding file in the test corpus. " That would definitely be doable. The os8-cp action file is totally your friend here!

I presume that an answer to this was gating on you getting the parallel make working. Is that finally ok?

These are my remaining punch list items:

1. Test out FORTRAN II and make sure it works.
2. Test all 4 tu56 images. -- Confirm IF=3 booting works for them all.
3. Consider including a test FORTRAN II Program.
4. Consider adding a tape option to replace FORTRAN II and BASIC with FORTRAN IV.

Q3: What other gaps remain?

We depend on OS8_BOOT_DISK, but RUNNING that disk touches it.

Yes, I can see how that might cause a problem.

put a vertical bar before $OS8_BOOT_DISK?

I've used that several places in the current Makefile, though I don't have a strong sense of when it can help. Give it a try and see if it changes the behavior in a sensible fashion.

If not, then maybe we should be building the disk to one location, then copy it to a different directory for make run to use, knowing the pristine copy is the one we should be installing.

Are we agreed that v3f-build.rk05 should move to obj, but that v3f-made.rk05 should stay in bin for installation as the "v3f source and build running playpen"?

You're running that part of the project, and you've got my opinion, so now you just need to decide.

...validation...mkos8...os8-cp...

I won't be writing any software to do binary comparisons of the new images to the old. That'll take way too long. (Educated guess: a month of solid CPU time on my quad-core iMac.)

At most, I may port tools/test-mkos8 to use os8-run instead. That'll take "only" about a week to regenerate the test corpus, the value of which is simply that it lets you check if the tool gives the same build twice, given the same inputs. For that to be useful, the inputs do in fact have to be the same. Are the inputs to os8-run now stable?

If that process starts throwing failures, then and only then I might look at an os8-cp based dissector to find the differences at a sub-medium level. As long as the generated media are identical, there's no point comparing individual files.

I presume that an answer to this was gating on you getting the parallel make working.

No, I want that for its own value: the build is many times faster on my development machines with a parallel build. That's inherently valuable, especially when building on a Pi 3.

Getting parallel builds working is also a good test of your dependencies: if they break under parallel builds, they're probably order-of-operations dependent with serial builds, too.

tools/test-mkos8 doesn't use parallel make, because it depends on the simulator and such already being built. Since it knows mkos8 is inherently single-threaded, it just runs multiple mkos8 builds in parallel, which is a wholly separate thing from parallel make.

Is that finally ok?

No, and I'm kind of mystified by the fact that it isn't, since I've given you a method to recreate the symptom. Here's the latest failure here:

Building OS/8 v3d tu56 image
for td12k configuration of TD8E.
At line 54, /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3d.rk05 must exist but was not found. Not mounting.

Fatal error encountered in ./media/os8/scripts/all-tu56.os8 at line 54:
	mount rk0 $bin/v3d.rk05 required
At line 58, /usr/local/src/pidp8i/os8-v3f-extensions/bin/v3d.rk05 must exist but was not found. Not mounting.

Fatal error encountered in ./media/os8/scripts/all-tu56.os8 at line 58:
	mount rk0 $bin/v3d.rk05 required scratch

I then did a fossil clean and retried, and this time I see that it's trying to run os8-run before the simulator is even fully built! Clearly the dependencies aren't correct yet.

Consider including a test FORTRAN II Program.

There's one already in the tree, if you just want to use it.

Consider adding a tape option to replace FORTRAN II and BASIC with FORTRAN IV.

That requires adding a medium-specific configuration option. So far, we have only feature-oriented options. They affect media differently only in that some media don't have all of the features.

I'd like to continue with the split where the TU56 tape media are stripped-down relative to the RK05 media. If someone wants OS/8 V3F with FORTRAN IV, they're a prime recruit for the future V3F-on-RK05 project.

What other gaps remain?

You appear to have not checked in file media/os8/3finit.tx.in.

(The above parallel make tests were done with checkin b17efa3579 backed out, else I couldn't have even gotten that far.)

I think I've added the necessary dependencies. See the checkin comment for the reason it's on a branch of your branch, rather than directly on it.

Unfortunately, I'm now getting yet another pexpect timeout error:

bin/os8-run ./media/os8/scripts/v3f-control.os8
...noisenoisenoise...
File "/usr/local/src/pidp8i/os8-v3f-extensions/lib/simh.py", line 203, in __init__
  'Failed to lock /dev/gpiomem', timeout = 3):
...noisenoisenoise...
searcher: searcher_re:
    0: re.compile("^PiDP-8/I [a-zA-Z].*\[.*\]")
    1: re.compile("Failed to lock /dev/gpiomem")

I've cut it down like this because I think what it's telling me is that it's expecting to find a "PiDP-8/I" banner line, and it is getting a complaint about /dev/gpiomem instead because this is a non-Pi platform. I suspect it's not finding that banner due to the missing 3finit.tx.in file.

What other gaps remain?

I just discovered three pidp8i-sim instances running in the background on this computer, presumably left behind by failed os8-run operations. (Symptom: ~90% CPU usage of this quad-core machine. :) ) There must be a way to modify the Python on-die behavior to shut the simulator down before exiting.

A bit of web searching says we probably need a high-level exception handler, but I don't think you want the catch clause to just exit the program. You probably want to do something like the default on-die handler, with the stack trace and all.

I merged in your changes because, on first reading them they seemed right, and on testing they seemed to behave similarly to what I had already seen.

However, now that I've done more testing I think they're not right.

Given that the timestamp OS8_BOOT_DISK changes every time you boot it, we have to depend on its existence, not on its age relative to the output files.

Adding a dependency on it to V3F_MADE_RK05, I think will force a redundant build, since that's already dependent on V3F_BUILD_RK05.

V3F_BUILD_RK05 should depend on the existence but not the timeliness of OS8_BOOT_DISK.

The two V3F tu56 images depend on V3F_MADE_RK05, which depends on V3F_BUILD_RK05, which depends on OS8_BUILD_DISK, so here too adding a dependency on it to V3F_MADE_RK05, I think will force a redundant build.

The two V3D tu56 images should depend on the existence of OS8_BOOT_DISK but not its timeliness.

I'm now doing a test run under mmake of this.

Please re-run your make with the new Makefile.in with my dependency corrections.

But I fear that the problem you encountered will still happen.

v3f-control.os8 uses v3d.rk05 to perform builds from source. It should not, in any way, depend on 3finit.tx.

The pexpect stuff has a timeout. I've never seen os8-run keep going past a missed pexpect. It always times out, and gives us our stack trace.

What are other possible explanations?

I think there was a subtle dependency bug on how I was creating/cleaning up v3f-build.rk05. I've moved it to the obj directory, and fixed how make clean was not cleaning up that file, which I think led to some problems on rebuilding.

I think this closes out the open issue regarding cleanup of the v3f build.

os8-cp now understands $bin, $obj, $os8mi, $os8mo expansions of POSIX paths, just like os8-run does.

In the light of all I have learned while writing os8-run, os8-cp looks pretty MESSY.

I'm switching over to using tools/mmake it's definitely a more robust test of building. I'm getting consistent success running that now. I'll send you a copy of my log files in private email.

Re: "Are the inputs to os8-run now stable?"

What do you mean by "inputs"?

• I've not changed the command language since I began this thread.
• Although I've made changes to the scripts that make the bootable tu56 images, the script that makes v3d.rk05, a.k.a. OS8_BOOT_DISK has not changed since 29 September.
• The script that makes v3d-dist.rk05 has not changed since 4 October when I modified the BUILD.SV image to have add td8e drivers TD8ESY, ROMMSY, TD8EA drivers included by default.
• The two scripts that cooperate to populate the 4 bootable tu56 images might change a bit more, as v3f work proceeds.

I believe the answer to your question is, YES.

I think the only blocker to the merge is getting my branch to consistently build for you, and whatever work you want to do switching tools/test-mkos8 over to os8-run. After I play a little with Fortran II testing, and confirming that this stuff works on my pi hardware, I may look at that stuff.

Initial tests are successful. Whether it's perfect yet I couldn't say, but it is better for certain.

I fed the FORTRAN II PEP001.FT program to v3d.rk05 and all 4 tu56 images and got the same results. I'm going to call FORTRAN II working.

There isn't actually a source file PEP001.FT anywhere in the tree. I had to create a file with a paste-in from the wiki page, which is ok, I guess.

Do you think I should PEP001.FT onto the packs somewhere? How would we prevent people from being confused about whether this was the FORTRAN II or FORTRAN IV version? (On the tapes, that's easy because currently FORTRAN IV isn't installed.)