Help Writing TSR Program(s) in NASM Assembly for DOS

Question

I've been trying to write TSR (Terminate-Stay-Resident) programs (in general) in Assembly (16-bit) for MS-DOS. I've read through a Wikipedia page on TSR and also a page on using it specifically in DOS (but it seems to be teaching it in C and not Assembly directly). I've looked at a site with tons of DOS interrupt documentation and find this one, this one, and another most relevant to TSR programs. I can't post all of the links because as a new user I can have up to 2 hyperlinks on a post.

So, I've tried writing a (seemingly) very simple TSR program in real mode flat model (.COM file format) in NASM. Here's the code:

[BITS 16]
[ORG 0x0100]

[SECTION .text]

Start:
; Get current interrupt handler for INT 21h
mov AX,3521h                ; DOS function 35h GET INTERRUPT VECTOR for interrupt 21h
int 21h                     ; Call DOS  (Current interrupt handler returned in ES:BX)

mov WORD [v21HandlerSegment],ES     ; Store the current INT 21h handler segment
mov WORD [v21HandlerOffset],BX      ; Store the current INT 21h handler offset

; Write new interrupt handler for INT 21h
mov AX,2521h                ; DOS function 25h SET INTERRUPT VECTOR for interrupt 21h
mov DX,TSRStart             ; Load DX with the offset address of the start of this TSR program
;   DS already contains the segment address, it is the same as CS in this .COM file
int 21h                     ; Override the INT 21h handler with this TSR program

; The TSR program will be called even when this portion uses INT 21h to terminate and stay resident
mov AX,3100h                ; DOS function TSR, return code 00h
mov DX,00FFh                ; I don't know how many paragraphs to keep resident, so keep a bunch
int 21h                     ; Call our own TSR program first, then call DOS

TSRStart:
push WORD [v21HandlerSegment]       ; Push the far address of the original 
push WORD [v21HandlerOffset]        ;   INT 21h handler onto the stack
retf                                ; Jump to it!


[SECTION .data]
v21HandlerSegment dw 0000h
v21HandlerOffset  dw 0000h

When I assemble this and execute it inside DOS, instead of returning back to the DOS prompt it hangs the system (no activity occurs except the hardware cursor just blinks below the last prompt). I guess memory garbage might be executing but you get the point.

Could anybody please help to either figure out what the problem with this code is and / or offer general advice for coding TSR's in DOS? Thanks in advance, any help is very much appreciated!

Laugh

Did I just enter a time warp and zip back 20 years? – Ariellearies

@Ariellearies Yep. Don't think I'm coding in this as a main language (I also code Java), I just need to know how to code a TSR in Assembly for demonstration purposes. – Laugh

Dude, you're doing programming archeology.. +1 for that! – Northnortheast

Answer 1

I figured it out. After looking through a couple more sources, I discovered that this code:

push WORD [v21HandlerSegment]       ; Push the far address of the original 
push WORD [v21HandlerOffset]        ;   INT 21h handler onto the stack

needs to be something like this:

push WORD [CS:v21HandlerSegment]       ; Push the far address of the original 
push WORD [CS:v21HandlerOffset]        ;   INT 21h handler onto the stack

because those memory references are referencing from the data segment, which isn't set up from the caller of the TSR. So basically I was referencing data from something else's data block...

This can also be accomplished by putting CS in DS (and then putting DS's original value back) like this:

push DS
push CS
pop DS
; Memory references....
pop DS

Laugh

Answer 2

1. You need to use a cs: segment override to access the TSR's data from within a general-purpose interrupt handler, because the ds value is an arbitrary user's register then.

2. You don't need to push the next handler's address onto the stack then jump with a retf. It is simpler to do jmp far [cs:...] (and that has a shorter encoding). But your method works fine too.

3. You can put your initialisation handling (not needed in the resident installed handler) at the end of your program image. This is a trivial optimisation of the TSR's size.

4. To calculate the size of your resident process, use NASM labels. To allow shift (or divide) operations needed to figure out a length in paragraphs, only use deltas of labels. A delta (difference) is a scalar value to NASM, so can be used in calculations. A (non-struc) label alone is not a scalar.

Here's an example using all of these:

        cpu 8086
        bits 16
        org 256

start:
        jmp init

        align 4
int21old:
        dd 0

int21handler:
        jmp far [cs:int21old]

end_of_resident:

init:
        mov ax, 3521h
        int 21h
        mov word [int21old + 2], es
        mov word [int21old], bx

        mov ax, 2521h
        mov dx, int21handler
        int 21h

        mov ax, 3100h
        mov dx, (end_of_resident - start + 256 + 15) >> 4
        int 21h

The size calculation computes the delta of two labels, adds in 256 for the process's PSP (same as the org 256), adds 15 to make the shift-division round up, then shifts down into an amount of paragraphs.

Unconnected

I'm the upvote. You could also encode a dummy JMP ptr16:ptr16 in the interrupt handler and use self modifying code in init to modify the segment and offset of the JMP. Then when interrupts occur the address to jump to is encoded in the instruction rather than requiring the jmp address to be read from another memory location. There will be a one time hit as the associated cache line is invalidated. – Meleager

@Michael Petch: If you do want to go this route (of Self Modifying Code), you can have NASM encode jmp 0:0 then add a line like int21old: equ $ - 4 immediately after the jump instruction. However, if you want to add IBM interrupt sharing protocol (or full AMIS) support to a TSR, you need to use jmp far [mem] because the interrupt sharing protocol header doesn't allow for the far jmp opcode. Refer to ctyme.com/intr/rb-4214.htm for the interrupt sharing protocol header. – Unconnected

You weren't doing interrupt multiplexing. My comment was in the context of the code you posed. And yes, when I said modify the JMP putting in a dummy JMP and simply modify the segment and the offset. Your method is what I'd do and in MASM you can use ORG to back the location counter up and place a label with a type there. – Meleager

@Michael Petch: Ah yes I misread and thought you meant the other way, which is db 0EAh \ dw 0, 0. And yes, I didn't use any IISP or AMIS, it's just a note on how actual programs might need the indirect jump. (Also something else I didn't do is actually optimising the size and behaviour, like re-using part of the PSP starting at offset 50h, allocating a new block for the resident portion, freeing the environment block, relocating the initialisation process, or freeing the process's file handles. Refer to hg.ulukai.org/ecm/rxansi for most of these.) – Unconnected

Yes, I think I've written an answer doing TSRs and relocating the resident code starting at offset 5ch in the PSP (80h if you aren't using FCB's in the interrupt handler (which is usually not the case). I don't usually start at 50h as many of the bytes are considered reserved so lord knows what any individual version of DOS is using them for. – Meleager

@Michael Petch: You can re-use memory at PSP:5Ch and also use FCBs later, just not FCBs at the two particular "default FCB" slots in the PSP. If I recall, opening the first default FCB overwrites part of the second default FCB actually; programs that want to use both are expected to copy them elsewhere into memory. – Unconnected

I should have been clear I was referring to the two FCBs in the PSP. Of course you aren't limited to them. I can't think of a likely scenario where that would be the case so I usually use 5ch rather than 80h. – Meleager

Was wondering why I had a hard time finding it. It was part of a comment discussion with Sep Roland and the code he posted as part of this answer: https://mcmap.net/q/1318089/-how-do-i-properly-hook-interrupt-28h-in-assembly-for-dos-and-restore-it . – Meleager