VSTE (VLAD Surface Tracing Engine) by Qark is an early attempt at an EPO (Entry-Point Obscuring) virus. VSTE is used in Qark's TraceVir. VSTE works very much like a length disassembler to trace the targeted file until a point is found to patch the victim.
VSTE takes as arguments a file handler to the victim file and a pointer to a temporary 300 byte buffer to read the file contents. VSTE returns a file pointer to the instruction to be patched and pointer to a buffer containing the first byte of that instruction on.
VSTE does not read the entire file into memory at once (that kind of memory is not always available under MS-DOS) but lseeked around the file, following the code and reading 300 bytes at a time. Files accessed by VSTE are treated as 'raw' with no formatting and the first instruction being at the very start of the file. This works for .COM files, but does not support other formats like MZ .EXE files.
VSTE only supports 8086 instructions and stops at anything 80286+. It is worth noting that VSTE
treats opcode 0Fh as 'POP CS' not an extended opcode, which is only the case with the 8086 (remember VSTE was released in 1996). Beside following instructions by their length, VSTE followed JMP/CALL NEAR and JMP SHORT instructions. Other than 80286+ instructions, VSTE also terminates on conditional jumps (07xh), JMP/CALL FAR and RET/IRET instructions.
Original research by JPanic aka @JPanicVX