Qwen3-VL-4.3B-semantic

An expanded-vocabulary variant of Qwen/Qwen3-VL-4B-Instruct, designed for use as a text encoder in Krea 2.

The original 36 decoder layers and vision tower are retained. The main architectural modification is an expanded embed_tokens / tied lm_head vocabulary containing additional whole-word English tokens.

https://huggingface.co/Sorenvza/Qwen3-VL-4_3B-semantic

What changed

| | Base | This model |

| ----------------- | -------: | ---------------------------------------: |

| Tokenizer length | 151,936 | 268,590 |

| Added word tokens | — | 116,654 |

| New parameters | — | 298,634,240 |

| Hidden size | 2,560 | 2,560 |

| Decoder layers | 36 | 36 |

| Vision tower | Original | Original |

| Main modification | — | Expanded embed_tokens / tied lm_head |Semantic initialization

The added embeddings were constructed in several stages:

1. English lexical candidates were selected from the external lexical resources described below.

2. Initial semantic representations were obtained from ConceptNet Numberbatch.

3. WordNet-derived hypernym relationships were used to impose additional lexical structure.

4. The semantic representations were aligned to the original Qwen embedding space using a linear least-squares projection based on the mean embeddings of the corresponding original subword representations.

5. The resulting vectors were used to initialize the added Qwen vocabulary rows.

This procedure should be understood as semantic embedding expansion and initialization, not as full pretraining or continued pretraining of the Qwen3-VL transformer.

Data and external resources

This model incorporates information derived from several external lexical and semantic resources.

ConceptNet Numberbatch

ConceptNet Numberbatch 19.08 was used as the initial semantic representation for the added vocabulary.

Source:

• ConceptNet

• ConceptNet Numberbatch

• Version: 19.08

The applicable ConceptNet / Numberbatch license and attribution requirements apply to the corresponding derived data.

WordNet / Open Multilingual WordNet

WordNet-derived lexical relations, including hypernym information, were used during semantic embedding construction.

The local lexical environment used the NLTK omw-1.4 resource where applicable.

The applicable WordNet / OMW licensing and attribution requirements apply to the corresponding derived data.

Wiktionary / Wiktextract

Wiktionary-derived lexical information was obtained through Wiktextract / Kaikki where applicable.

The exact dump or snapshot used to construct the released lexical data should be recorded for reproducibility.

Wiktionary-derived data is subject to the applicable Wiktionary licensing terms, including CC BY-SA and GFDL where applicable.