Multimodal and Hierarchical Topic Modeling

[juxiepao]

Dear Maarten Grootendorst,

I have a question regarding the integration of Hierarchical Topic Modeling after applying Multimodal Topic Modeling. In my project, I have trained the Multimodal Topic Model using both text and image data. I would like to use Hierarchical Topic Modeling as an indicator to merge the results from the multimodal model.

As far as I understand, Hierarchical Topic Modeling is based on c-TF-IDF, which uses only the textual documents. My question is: how can I reflect the results from the images in this process? Would it be possible to base the hierarchical modeling on embeddings, or is this method strictly limited to text?

My goal is to combine both textual and visual information when creating the hierarchical structure. Any guidance on how to achieve this would be greatly appreciated. Any advice on how to proceed with incorporating image results into the hierarchical model would be greatly appreciated.

Thank you for your time, and I look forward to your guidance.

[MaartenGr]

It should be possible to use the embeddings to perform hierarchical modeling. I should note though that hierarchical topic modeling will not combine images at the moment and is meant for textual representations instead. Therefore, it will only work if you used both text and image data during training since you would need to give it the text.

Lastly, you might need to install BERTopic from the main repo to enable embeddings for hierarchical topic modeling (although I believe it might already be in the latest release).

[juxiepao]

Dear Maarten Grootendorst,

Thank you so much for your quick response. I'm a bit confused about something. I reinstalled BERTopic in Colab from the main repository, and when I tried to provide embeddings to the hierarchical_topics() function, I encountered an error stating that it received an unexpected keyword argument 'embeddings'. The embeddings I trained are based on both text and images.

# Perform Hierarchical Topic Modeling hierarchical_topics = topic_model.hierarchical_topics(docs=texts_truncated, embeddings=doc_image_embeddings) TypeError: BERTopic.hierarchical_topics() got an unexpected keyword argument 'embeddings'

PS: I have one more question. If I train the embeddings using only images, what do the text representations of the topics represent? Is this information extracted from or generated based on the images? Additionally, if I train the embeddings using both text and images, are the visualized text representations based solely on the text, or are they influenced by both the textual and image data?

Thank you again for your assistance. I look forward to your insights.

[MaartenGr]

You would need to use use_ctfidf=False instead as it uses the topic embeddings and not the doc_image_embeddings.

If I train the embeddings using only images, what do the text representations of the topics represent? Is this information extracted from or generated based on the images?

They are the captions that were generated from the most representative images. You can find more about that here.

Additionally, if I train the embeddings using both text and images, are the visualized text representations based solely on the text, or are they influenced by both the textual and image data?

The text itself is based solely on the textual data and not the images. The clustering however depends on the embeddings that you used (which can be document embeddings, image embeddings, or both).

[juxiepao]

Thanks again for your help. I’ve found that I should be using the use_ctfidf function. However, after applying it, I noticed that the results don't seem to differ from when I didn’t use this option. What could be the problem?

Here is the way how I trained the embedings and topic models

model = MultiModalBackend('clip-ViT-B-32', batch_size=32) doc_image_embeddings = model.embed(texts_truncated, image_paths) topic_model = BERTopic(embedding_model=model,vectorizer_model=vectorizer_model,representation_model=representation_model, calculate_probabilities=True, verbose=True, min_topic_size=14) topics, probs = topic_model.fit_transform(documents=texts_truncated,images=image_paths,embeddings=doc_image_embeddings)

Here use_ctfidf = True
`# Perform Hierarchical Topic Modeling
hierarchical_topics_1 = topic_model.hierarchical_topics(texts_truncated,use_ctfidf=True)

Visualize the hierarchy

fig_1 = topic_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics_1)
fig_1.show()`

Here use_ctfidf = False
`# Perform Hierarchical Topic Modeling
hierarchical_topics_2 = topic_model.hierarchical_topics(texts_truncated,use_ctfidf=False)

Visualize the hierarchy

fig_2 = topic_model.visualize_hierarchy(hierarchical_topics=hierarchical_topics_2)
fig_2.show()`

[MaartenGr]

However, after applying it, I noticed that the results don't seem to differ from when I didn’t use this option. What could be the problem?

I don't think there is a problem. I believe it automatically selects what is available. Since c-TF-IDF isn't, it will use the embedding approach instead. Therefore, results stay the same.