assembler: wrap Module's with Arc and update Tokeniser (still WIP)

Implements complete tokenizer with Arc-wrapped modules

Enhances module handling by wrapping Module instances in Arc for thread-safe sharing across the assembler pipeline.

Implements full tokenization logic with pattern matching for all token types including labels, registers, immediates, directives, instructions, symbols, and strings.

Adds comma token support and proper EOF handling to complete the lexical analysis phase.

Generated AI slop commit message, may not be super accurate or it may be a bit too serious lol.
This commit is contained in:
2025-06-25 17:35:03 +01:00
parent 27267e3daa
commit 9b9e153500
3 changed files with 172 additions and 11 deletions
+3 -1
View File
@@ -23,8 +23,10 @@ pub enum TokenType {
Instruction(InstructionToken),
/// Label definition (e.g., `loop_start:`).
Label(LabelToken),
/// Assembler directive (e.g., `.global`, `.section`, `.dw`).
/// Assembler directive (e.g., `.global`, `.section`, `.dw`, `.resb`).
Directive(DirectiveToken),
/// Comma separator.
Comma,
/// End of line.
Newline,
/// End of file.